@reticlehq/server 2.7.0 → 2.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +14 -5
- package/SKILL.md +64 -29
- package/dist/capsule/causal-summary.d.ts +20 -0
- package/dist/capsule/causal-summary.js +31 -6
- package/dist/capsule/divergence.js +7 -1
- package/dist/cli/chromium-hint.d.ts +33 -5
- package/dist/cli/chromium-hint.js +82 -12
- package/dist/cli/cli-doctor.js +28 -25
- package/dist/cli/cli-kill.d.ts +58 -0
- package/dist/cli/cli-kill.js +162 -0
- package/dist/cli/cli-launch.d.ts +17 -0
- package/dist/cli/cli-launch.js +28 -0
- package/dist/cli/cli-parse.d.ts +16 -1
- package/dist/cli/cli-parse.js +32 -9
- package/dist/cli/cli-port.d.ts +8 -0
- package/dist/cli/cli-port.js +16 -0
- package/dist/cli/cli-verify.js +2 -2
- package/dist/cli/doctor-sessions-line.d.ts +12 -0
- package/dist/cli/doctor-sessions-line.js +66 -0
- package/dist/cli/port-holder.d.ts +10 -0
- package/dist/cli/port-holder.js +26 -2
- package/dist/cli.js +99 -14
- package/dist/crawl/crawl-tools.js +12 -2
- package/dist/crawl/crawl.d.ts +13 -0
- package/dist/crawl/crawl.js +50 -2
- package/dist/events/contradictions.d.ts +14 -0
- package/dist/events/contradictions.js +26 -17
- package/dist/events/predicate-eval.d.ts +20 -1
- package/dist/events/predicate-eval.js +125 -3
- package/dist/events/predicate.js +170 -18
- package/dist/flows/flow-classify.js +1 -1
- package/dist/flows/flow-tools.js +12 -4
- package/dist/flows/heal-run.js +1 -1
- package/dist/honesty/blind-spots.js +6 -2
- package/dist/honesty/unsettled.d.ts +40 -0
- package/dist/honesty/unsettled.js +78 -0
- package/dist/honesty/verified.d.ts +7 -0
- package/dist/honesty/verified.js +3 -3
- package/dist/index.js +3 -0
- package/dist/init/agent-rules.d.ts +16 -0
- package/dist/init/agent-rules.js +96 -22
- package/dist/init/astro-patch.d.ts +2 -1
- package/dist/init/astro-patch.js +11 -5
- package/dist/init/capabilities.d.ts +22 -0
- package/dist/init/capabilities.js +76 -0
- package/dist/init/confirm.d.ts +96 -0
- package/dist/init/confirm.js +138 -0
- package/dist/init/cra.js +3 -2
- package/dist/init/non-js-project.js +29 -5
- package/dist/init/plan-framework.js +53 -15
- package/dist/init/plan.d.ts +40 -4
- package/dist/init/plan.js +163 -40
- package/dist/init/run.d.ts +24 -2
- package/dist/init/run.js +148 -42
- package/dist/init/slash-command.d.ts +1 -1
- package/dist/init/slash-command.js +46 -6
- package/dist/init/snippets.d.ts +44 -4
- package/dist/init/snippets.js +161 -37
- package/dist/init/vite-config.js +29 -3
- package/dist/input/network-mock-tools.d.ts +19 -0
- package/dist/input/network-mock-tools.js +29 -4
- package/dist/license/license.js +1 -1
- package/dist/mcp/mcp-proxy.d.ts +69 -3
- package/dist/mcp/mcp-proxy.js +205 -20
- package/dist/mcp/mcp.js +23 -7
- package/dist/mcp/proxy-lifecycle.d.ts +1 -1
- package/dist/mcp/proxy-lifecycle.js +1 -1
- package/dist/pool/browser-pool.d.ts +22 -2
- package/dist/pool/browser-pool.js +44 -4
- package/dist/project/learned-routes.d.ts +24 -0
- package/dist/project/learned-routes.js +63 -0
- package/dist/project/project-store.d.ts +6 -0
- package/dist/project/project-store.js +25 -1
- package/dist/session/no-session-diagnosis.d.ts +19 -0
- package/dist/session/no-session-diagnosis.js +71 -18
- package/dist/session/no-session-watch.js +12 -0
- package/dist/session/ref-provenance.d.ts +54 -0
- package/dist/session/ref-provenance.js +61 -0
- package/dist/session/session-lease.js +1 -1
- package/dist/telemetry/automation-hint.d.ts +17 -0
- package/dist/telemetry/automation-hint.js +71 -0
- package/dist/telemetry/bug-found.d.ts +0 -20
- package/dist/telemetry/bug-found.js +38 -21
- package/dist/telemetry/cli-telemetry.js +7 -1
- package/dist/telemetry/feedback.js +1 -1
- package/dist/telemetry/init-telemetry.js +9 -1
- package/dist/telemetry/install-source.d.ts +36 -0
- package/dist/telemetry/install-source.js +40 -0
- package/dist/telemetry/run-telemetry.js +4 -1
- package/dist/telemetry/session-metrics.js +5 -0
- package/dist/telemetry/telemetry.d.ts +5 -1
- package/dist/telemetry/telemetry.js +15 -2
- package/dist/telemetry/tool-refused.d.ts +19 -0
- package/dist/telemetry/tool-refused.js +55 -0
- package/dist/telemetry/verification-of.js +46 -7
- package/dist/temp-dir.d.ts +27 -0
- package/dist/temp-dir.js +46 -0
- package/dist/tools/act-tools.js +45 -1
- package/dist/tools/assert-grade.d.ts +1 -1
- package/dist/tools/assert-grade.js +27 -5
- package/dist/tools/assert-verdict.js +18 -1
- package/dist/tools/contract-tools.js +4 -0
- package/dist/tools/dynamic-tools.js +1 -1
- package/dist/tools/error-recovery.d.ts +10 -0
- package/dist/tools/error-recovery.js +64 -4
- package/dist/tools/feedback-invite.js +1 -1
- package/dist/tools/invoke-tool.js +139 -23
- package/dist/tools/lease-tools.d.ts +29 -1
- package/dist/tools/lease-tools.js +58 -5
- package/dist/tools/navigate-result.js +1 -1
- package/dist/tools/observe-tools.js +1 -1
- package/dist/tools/observed-nothing.js +1 -1
- package/dist/tools/react-grace.d.ts +48 -0
- package/dist/tools/react-grace.js +87 -0
- package/dist/tools/read-tools.js +20 -4
- package/dist/tools/settle-in-flight.d.ts +79 -0
- package/dist/tools/settle-in-flight.js +106 -0
- package/dist/tools/snapshot-delta.d.ts +7 -0
- package/dist/tools/snapshot-delta.js +130 -6
- package/dist/tools/tool-surface.js +19 -0
- package/dist/tools/tools.js +16 -4
- package/docs/README.md +56 -11
- package/docs/actions.mdx +87 -0
- package/docs/agent-cheatsheet.md +93 -29
- package/docs/architecture.md +46 -35
- package/docs/benchmarks.md +76 -72
- package/docs/best-practices.mdx +211 -0
- package/docs/capabilities.mdx +387 -0
- package/docs/cli/affected.mdx +51 -0
- package/docs/cli/capsules.mdx +69 -0
- package/docs/cli/cloud.mdx +127 -0
- package/docs/cli/doctor.mdx +76 -0
- package/docs/cli/drive.mdx +55 -0
- package/docs/cli/feedback.mdx +67 -0
- package/docs/cli/gate.mdx +64 -0
- package/docs/cli/help.mdx +92 -0
- package/docs/cli/hunt.mdx +57 -0
- package/docs/cli/identify.mdx +62 -0
- package/docs/cli/init.mdx +94 -0
- package/docs/cli/kill.mdx +91 -0
- package/docs/cli/license.mdx +48 -0
- package/docs/cli/mcp.mdx +61 -0
- package/docs/cli/open.mdx +68 -0
- package/docs/cli/restart.mdx +60 -0
- package/docs/cli/rollback.mdx +35 -0
- package/docs/cli/serve.mdx +62 -0
- package/docs/cli/status.mdx +64 -0
- package/docs/cli/stop.mdx +56 -0
- package/docs/cli/telemetry.mdx +57 -0
- package/docs/cli/update.mdx +52 -0
- package/docs/cli/verify.mdx +57 -0
- package/docs/cli/version.mdx +39 -0
- package/docs/cli/watch.mdx +55 -0
- package/docs/cli.mdx +219 -0
- package/docs/debugging.md +13 -9
- package/docs/deploy-checks.md +15 -11
- package/docs/desktop-apps.md +49 -40
- package/docs/desktop.mdx +188 -0
- package/docs/docs.json +363 -0
- package/docs/enterprise.md +20 -16
- package/docs/faq.mdx +88 -0
- package/docs/fixtures.md +11 -7
- package/docs/flows.md +26 -22
- package/docs/for-agents.md +71 -0
- package/docs/frameworks.mdx +199 -0
- package/docs/gate-plan.md +45 -41
- package/docs/gates.md +22 -18
- package/docs/getting-started.md +76 -65
- package/docs/human-control.md +22 -18
- package/docs/index.mdx +101 -0
- package/docs/install-agentic.mdx +215 -0
- package/docs/install-manual.mdx +194 -0
- package/docs/instrumentation.mdx +232 -0
- package/docs/integration-patterns.md +38 -34
- package/docs/local-registry.md +16 -12
- package/docs/multi-agent-testing.md +16 -12
- package/docs/packages/babel-plugin.mdx +70 -0
- package/docs/packages/browser.mdx +111 -0
- package/docs/packages/core.mdx +106 -0
- package/docs/packages/electron.mdx +87 -0
- package/docs/packages/eslint-plugin.mdx +84 -0
- package/docs/packages/next.mdx +91 -0
- package/docs/packages/react.mdx +91 -0
- package/docs/packages/server.mdx +98 -0
- package/docs/packages/tauri.mdx +63 -0
- package/docs/packages/test.mdx +108 -0
- package/docs/packages/vite-plugin.mdx +96 -0
- package/docs/packages.mdx +217 -0
- package/docs/platform-integration.md +33 -26
- package/docs/predicates.mdx +179 -0
- package/docs/quickstart.mdx +211 -0
- package/docs/recipes.mdx +242 -0
- package/docs/skill-file.mdx +105 -0
- package/docs/state-management.mdx +197 -0
- package/docs/style.css +14 -0
- package/docs/system-map.md +47 -32
- package/docs/telemetry-contract.md +93 -50
- package/docs/telemetry-events.mdx +430 -0
- package/docs/telemetry.md +57 -52
- package/docs/testing.md +10 -6
- package/docs/token-efficiency.md +30 -26
- package/docs/tools-act-and-wait.mdx +123 -0
- package/docs/tools-act-sequence.mdx +120 -0
- package/docs/tools-act.mdx +82 -0
- package/docs/tools-assert.mdx +70 -0
- package/docs/tools-console.mdx +59 -0
- package/docs/tools-inspect.mdx +85 -0
- package/docs/tools-navigate.mdx +63 -0
- package/docs/tools-network.mdx +86 -0
- package/docs/tools-observe.mdx +89 -0
- package/docs/tools-overview.mdx +157 -0
- package/docs/tools-query.mdx +82 -0
- package/docs/tools-session-and-feedback.mdx +91 -0
- package/docs/tools-sessions.mdx +65 -0
- package/docs/tools-snapshot.mdx +75 -0
- package/docs/tools-state.mdx +81 -0
- package/docs/tools-tools-and-run.mdx +125 -0
- package/docs/tools-wait-for.mdx +66 -0
- package/docs/troubleshooting.mdx +193 -0
- package/docs/usage.md +175 -153
- package/docs/vs-chrome-devtools-mcp.mdx +82 -0
- package/docs/vs-playwright-mcp.mdx +85 -0
- package/docs/vs-screenshots.mdx +96 -0
- package/docs/what-is-reticle.mdx +63 -0
- package/docs/why-reticle.mdx +107 -0
- package/package.json +2 -2
- package/docs/matrix/2.5.0/claude-code-machine.json +0 -19
- package/docs/matrix/2.5.0/codex-machine.json +0 -20
- package/docs/matrix/2.5.0/cursor-machine.json +0 -22
- package/docs/matrix/2.5.0/gemini-machine.json +0 -22
- package/docs/matrix/2.5.0/opencode-machine.json +0 -22
- package/docs/matrix/2.5.0/vscode-machine.json +0 -22
- package/docs/matrix/2.5.0/windsurf-machine.json +0 -22
- package/docs/matrix/MATRIX.md +0 -26
- package/docs/matrix/README.md +0 -59
- package/docs/mint.json +0 -61
|
@@ -1,12 +1,16 @@
|
|
|
1
|
-
|
|
1
|
+
---
|
|
2
|
+
title: Telemetry contract
|
|
3
|
+
description: 'Read this before adding a tool, an event, a finding kind, or a failure path. Telemetry fails silently.'
|
|
4
|
+
icon: file-contract
|
|
5
|
+
---
|
|
2
6
|
|
|
3
|
-
> For anyone
|
|
7
|
+
> For anyone (human or agent) adding a tool, an event, a finding kind, or a failure path to Reticle.
|
|
4
8
|
>
|
|
5
9
|
> The rules here are enforced by `packages/server/src/telemetry/telemetry-contract.test.ts`. If you break one, that test tells you which and where. This page is why.
|
|
6
10
|
|
|
7
11
|
## Why this has its own contract
|
|
8
12
|
|
|
9
|
-
**Telemetry fails silently.** Nothing throws when an event is missed. No test goes red. No user complains. The data is simply, permanently absent
|
|
13
|
+
**Telemetry fails silently.** Nothing throws when an event is missed. No test goes red. No user complains. The data is simply, permanently absent, and you find out months later when someone asks a question the data cannot answer, about a period you can never re-collect.
|
|
10
14
|
|
|
11
15
|
That has already happened here twice, and both times the code looked correct:
|
|
12
16
|
|
|
@@ -19,11 +23,11 @@ So the rule is not "remember to add telemetry". The rule is that the guard lives
|
|
|
19
23
|
|
|
20
24
|
### 1. Everything routes through a chokepoint
|
|
21
25
|
|
|
22
|
-
Tool usage, timing, errors, verifications and bugs are all recorded in **one place
|
|
26
|
+
Tool usage, timing, errors, verifications and bugs are all recorded in **one place**: `runTool` in `tools/invoke-tool.ts`. Adding a tool to `TOOLS` is all it takes to be instrumented.
|
|
23
27
|
|
|
24
28
|
Do **not** add telemetry inside a tool handler. If you find yourself wanting to, the metric probably belongs at the chokepoint, read off the result.
|
|
25
29
|
|
|
26
|
-
> The one exception is a path that genuinely does not go through `runTool
|
|
30
|
+
> The one exception is a path that genuinely does not go through `runTool`. Today that is only the verification runner (`reticle verify`, the HTTP verify surface), which has its own reporter in `telemetry/run-telemetry.ts`. **If you add a second dispatch path, it needs the same treatment**, and until it has one it is invisible. That gap existed for real: CI-found bugs were uncounted.
|
|
27
31
|
|
|
28
32
|
### 2. Names say what happened
|
|
29
33
|
|
|
@@ -31,20 +35,20 @@ Do **not** add telemetry inside a tool handler. If you find yourself wanting to,
|
|
|
31
35
|
|
|
32
36
|
## Counting defects: instances vs distinct
|
|
33
37
|
|
|
34
|
-
`bug_found` fires once per OCCURRENCE. A defect hit five times in a session is five events, which is the right raw signal
|
|
38
|
+
`bug_found` fires once per OCCURRENCE. A defect hit five times in a session is five events, which is the right raw signal; frequency is what says which classes of defect actually cost anybody anything. But it means a naive count answers "how often were defects hit", not "how many defects were found", while looking like it answers the second.
|
|
35
39
|
|
|
36
40
|
So every `bug_found` carries **`repeat`**: false the first time a KIND is seen in a session, true after. Count `repeat: false` for **distinct defects**; count everything for **instances**. Measured on a real app, the instance count was more than double the defect count. Publishing instances as defects inflates the claim accordingly.
|
|
37
41
|
|
|
38
42
|
The denominator is **`verification_completed`**, which fires per verdict with `via`, `verified`, `passed` and `falseGreenCaught`. Defects per verification is the honest rate; raw defect counts grow with usage and say nothing on their own.
|
|
39
43
|
|
|
40
|
-
**And `repeat` only means anything if the session remembers.** `SessionMetrics.reset()` runs at every periodic flush and used to clear the seen-kinds set with the window counters
|
|
44
|
+
**And `repeat` only means anything if the session remembers.** `SessionMetrics.reset()` runs at every periodic flush and used to clear the seen-kinds set with the window counters, so the same defect, re-found after a flush, reported `repeat: false` again. Sessions in the data run to 11.5 hours. Window counters zero on a flush; session-lifetime memory does not. (`session-window.test.ts`)
|
|
41
45
|
|
|
42
46
|
Two rules follow, and both are gated:
|
|
43
47
|
|
|
44
48
|
- **`repeat` is set at the EMISSION site, never by the classifier.** `bugsInResult` is a pure function over one tool result and cannot know what a session has already seen; if it ever grows a `repeat` field it will be guessing, and the guess becomes the published number. (`telemetry-contract.test.ts`)
|
|
45
|
-
- **Session-scoped, and it cannot be otherwise.** The payload carries no selector, URL or app detail by design, so the same defect in two sessions is unrecognisable as one
|
|
49
|
+
- **Session-scoped, and it cannot be otherwise.** The payload carries no selector, URL or app detail by design, so the same defect in two sessions is unrecognisable as one, and making it recognisable would require collecting exactly what this event refuses to collect.
|
|
46
50
|
|
|
47
|
-
The old set failed this so badly it confused its own authors
|
|
51
|
+
The old set failed this so badly it confused its own authors: `invoke` meant "the CLI ran" while `tool` meant "a tool was called", which is the opposite of how both read. A name that has to be looked up is a name that gets misread on a dashboard a year from now.
|
|
48
52
|
|
|
49
53
|
### 3. Names, never values
|
|
50
54
|
|
|
@@ -65,7 +69,7 @@ There is one narrow exception, and it is explicit rather than heuristic: paramet
|
|
|
65
69
|
If a set of kinds already exists in `@reticlehq/core`, **import it**. Do not re-list it.
|
|
66
70
|
|
|
67
71
|
```ts
|
|
68
|
-
// ✗ correct today, wrong the moment core gains a member
|
|
72
|
+
// ✗ correct today, wrong the moment core gains a member, and silent about it
|
|
69
73
|
const KINDS = new Set(['ui-advanced-request-failed', 'signal-contradicted' /* …10 more */]);
|
|
70
74
|
|
|
71
75
|
// ✓ cannot drift
|
|
@@ -78,40 +82,41 @@ A copied enum is a drift hazard anywhere. It is a **correctness** hazard when th
|
|
|
78
82
|
|
|
79
83
|
Every send is wrapped and best-effort. A telemetry failure must not fail a tool call, a verification, a daemon start, or `reticle init`.
|
|
80
84
|
|
|
81
|
-
The single exception is `daemon_stopped`, which is **awaited
|
|
85
|
+
The single exception is `daemon_stopped`, which is **awaited**, because the process exits immediately after and the send would otherwise be killed. Even then a failure resolves rather than throws.
|
|
82
86
|
|
|
83
87
|
## The event kinds, all of them
|
|
84
88
|
|
|
85
|
-
`TelemetryEventKind` in `@reticlehq/core` is the closed list. Seven of these went undocumented here for months
|
|
89
|
+
`TelemetryEventKind` in `@reticlehq/core` is the closed list. Seven of these went undocumented here for months (the doc described only the session-lifecycle half), so anyone building a dashboard from this page could not know that the transport and install events existed at all. A kind nobody documents is a kind nobody queries, and telemetry that nobody queries is telemetry nobody notices has stopped arriving. `telemetry-contract.test.ts` now fails when a kind is missing from this table.
|
|
86
90
|
|
|
87
91
|
| Kind | When | What it answers |
|
|
88
92
|
| --- | --- | --- |
|
|
89
93
|
| `reticle_installed` | first-ever run on a machine | install count, and the new-user curve |
|
|
90
|
-
| `cli_command_run` | a human ran a `reticle` subcommand | human intent
|
|
94
|
+
| `cli_command_run` | a human ran a `reticle` subcommand | human intent: `verify`/`gate` mean something very different from `status`. Never emitted for the internal `_daemon` spawn |
|
|
91
95
|
| `daemon_started` | the daemon came up | active sessions, DAU/WAU/MAU |
|
|
92
|
-
| `daemon_stopped` | clean exit | the rich session roll-up. **Count sessions with this one
|
|
96
|
+
| `daemon_stopped` | clean exit | the rich session roll-up. **Count sessions with this one**; see below |
|
|
93
97
|
| `session_progress` | periodic flush from a LIVE daemon | same payload, `final: false`. Sum work across both |
|
|
94
98
|
| `verification_completed` | a verdict was produced | the product's reason to exist: was an app actually verified |
|
|
95
|
-
| `project_profiled` | once per daemon start | stack, size, and how deeply the feature surface is used
|
|
99
|
+
| `project_profiled` | once per daemon start | stack, size, and how deeply the feature surface is used (activation vs retention) |
|
|
96
100
|
| `version_changed` | the running version differs from last seen | upgrade adoption, and whether a nudge caused it |
|
|
97
101
|
| `runtime_crashed` | an unhandled failure in the daemon | stability |
|
|
98
102
|
| `feedback_submitted` | `reticle feedback`, or an agent's report | the qualitative channel |
|
|
99
103
|
| `identified` | `reticle identify` | joins anonymous machine ids to a person who volunteered one |
|
|
100
104
|
| `mcp_client_connected` | an MCP client attached | how many sessions are agent-driven at all |
|
|
101
|
-
| `app_instrumented` | the first app carrying the SDK reached this daemon | **the funnel step everything turns on
|
|
105
|
+
| `app_instrumented` | the first app carrying the SDK reached this daemon | **the funnel step everything turns on**; see below |
|
|
102
106
|
| `mcp_connection_lost` | the proxy lost its daemon | **the transport-stability metric.** The disconnect that makes a user reopen `/mcp` is invisible without it |
|
|
103
107
|
| `init_completed` | `reticle init` finished | does install actually work, outside the fixtures gate |
|
|
104
108
|
| `bug_found` | a defect was detected in the app under test | the value delivered, as opposed to the work done |
|
|
109
|
+
| `tool_refused` | a tool could not do what was asked | WHY the largest cohort in the funnel goes quiet. See below |
|
|
105
110
|
|
|
106
|
-
## The install has two halves
|
|
111
|
+
## The install has two halves: `app_instrumented`
|
|
107
112
|
|
|
108
113
|
Reticle is only usable when both halves are done: the MCP server is registered so the agent has the tools, and the SDK is loaded by a running page so there is something for those tools to look at. They are done by different commands, at different times, often in different directories. Almost everyone completes the first. The second is where the users go.
|
|
109
114
|
|
|
110
|
-
Nothing measured the second. `daemon_started` and `mcp_client_connected` describe the agent half. `session_appConnects` describes the app half but is a **window counter
|
|
115
|
+
Nothing measured the second. `daemon_started` and `mcp_client_connected` describe the agent half. `session_appConnects` describes the app half but is a **window counter**: it resets on every flush, so a user whose app connected in one window reads zero in every other. The population it under-counts is precisely the population being measured, and a funnel built on it reported fewer instrumented users than there were users calling tools, which is impossible on its face.
|
|
111
116
|
|
|
112
117
|
`app_instrumented` fires **once per daemon run**, on the first session-ready only, so `daemon_started` → `app_instrumented` is a rate rather than an inference and a reloading page cannot inflate it. It carries `initialized` (had `init` run here), `agentAttached` (was an agent already waiting), and `msToFirstApp` (how long the daemon sat with nothing wired). It deliberately carries no stack and no framework: `project_profiled` already reports both for the same run, and the two join on `sessionId`.
|
|
113
118
|
|
|
114
|
-
What it still cannot see is **why** an app never connected
|
|
119
|
+
What it still cannot see is **why** an app never connected. Every cause for that is page-side (the non-localhost gate, a port mismatch, a stale build, a dev server never restarted), where the daemon has no visibility. That needs the SDK to report its own refusals, and it is the next thing to build.
|
|
115
120
|
|
|
116
121
|
## Sessions: `daemon_stopped` vs `session_progress`
|
|
117
122
|
|
|
@@ -119,49 +124,87 @@ Count sessions with **`daemon_stopped`** (`final: true`). It fires once, at a cl
|
|
|
119
124
|
|
|
120
125
|
A running daemon rolls its window up every 5 minutes as **`session_progress`** (`final: false`), same payload shape. Sum work across both; count sessions with neither summed nor doubled.
|
|
121
126
|
|
|
122
|
-
This split exists because the flush used to be emitted AS `daemon_stopped
|
|
127
|
+
This split exists because the flush used to be emitted AS `daemon_stopped`, an event named for an exit, fired while the process was alive. One day's export: 98 `daemon_stopped` events were 73 exits + 25 flushes, so a session count was 34% high. Worse, the two populations are opposites: every one of the 25 flushes had tool calls and **not one of the 73 exits did**, because a daemon that has served a tool never idle-exits and so never reaches a clean shutdown. A funnel over the raw event describes active sessions at one end and abandoned ones at the other.
|
|
123
128
|
|
|
124
129
|
The flush interval is also the **bound on what is lost**: nothing calls shutdown when a working daemon is finally killed, so its last partial window dies with it. At 30 minutes against a median 28-minute session that was most of the session. Only non-empty windows emit, so a short interval costs nothing on the daemons that never serve a tool.
|
|
125
130
|
|
|
126
131
|
## The session summary's newer fields
|
|
127
132
|
|
|
128
|
-
Four counters and one flag were added because the data could not answer questions we were already asking. All are properties on events that already exist
|
|
133
|
+
Four counters and one flag were added because the data could not answer questions we were already asking. All are properties on events that already exist (no new kinds), and all four counters are **omitted rather than sent as zero**, so a field's presence is itself the signal.
|
|
129
134
|
|
|
130
135
|
| Field | On | Means |
|
|
131
136
|
| --- | --- | --- |
|
|
132
|
-
| `noSessionErrors` | session summary | tool calls that failed because there was no app to reach
|
|
137
|
+
| `noSessionErrors` | session summary | tool calls that failed because there was no app to reach: no session, no session by that id, or several with none named. The largest drop-off in the funnel; it was previously reachable only by unpacking `errors[]`. |
|
|
133
138
|
| `consecutiveRepeats` | session summary | longest back-to-back run per tool name. `toolCounts` reports five useful calls and five retries of one failing call identically, and those are opposite facts. |
|
|
134
|
-
| `abandonedActions` | session summary | actions driven with no verdict AFTER them
|
|
135
|
-
| `
|
|
136
|
-
| `verification.
|
|
139
|
+
| `abandonedActions` | session summary | actions driven with no verdict AFTER them (the trailing unsettled run, not `actions - verifications`). That difference ignores order, so a verdict that drove nothing (a `flow_verify` over saved flows) silently paid for an abandoned action elsewhere. |
|
|
140
|
+
| `endedWithVerdict` | session summary (final only) | did this session ever produce a verdict. The headline metric, and previously the only thing in the payload that had to be COMPUTED, from lifetime counters sitting next to windowed ones, which is a subtraction that gets read wrong. Sent as `false` rather than omitted: a session that drove an app and never asked whether it worked is the finding. |
|
|
141
|
+
| `verification.browser` | `verification_completed` | `headless` \| `headed` \| `attached`: who DROVE the browser. `attached` (Reticle launched nothing, the SDK connected from a browser somebody else opened) is the common case in production, so on its own this is mostly "somebody's own browser". |
|
|
142
|
+
| `verification.brand` | `verification_completed` | WHICH browser it was: `chrome` \| `edge` \| `arc` \| `dia` \| `brave` \| `opera` \| `firefox` \| `safari` \| `other`, the closed `BrowserBrand` list in core. The axis `engine` cannot answer, since Chrome, Edge, Arc, Dia and Brave are all `blink`. The SDK reads `navigator.userAgentData.brands` (and the UA string on Firefox/Safari, which expose no `userAgentData`) and normalises IN THE PAGE: a raw brand or UA string is unbounded and fingerprintable and never leaves. Anything unrecognised is `other`. **Omitted rather than `"unknown"`** when the page did not say. A desktop webview has no brand and an older SDK does not report one, and a guess is indistinguishable from a measurement on a dashboard. |
|
|
137
143
|
| `verification.reason` | `verification_completed` | WHICH clause of `decideVerified` produced the verdict, from core's closed `VerifiedReason`: `inconclusive` \| `observation_lost` \| `assertion_failed` \| `contradicted` \| `already_true` \| `unclean_capture` \| `vacuous_grade` \| `outcome_pending` \| `outcome_unread` \| `unsettled` \| `proved`. See below. |
|
|
138
|
-
| `verification.uncleanLoss` | `verification_completed` | WHAT was lost when `reason` is `unclean_capture`, from core's closed `CaptureLoss`: `buffer_loss` (our server ring buffer evicted evidence from the window) \| `transport_gap` (our browser queue overflowed) \| `blind_spot` (a boundary in the page
|
|
139
|
-
| `bug.attribution` | `bug_found` | `app` \| `request` \| `reticle
|
|
144
|
+
| `verification.uncleanLoss` | `verification_completed` | WHAT was lost when `reason` is `unclean_capture`, from core's closed `CaptureLoss`: `buffer_loss` (our server ring buffer evicted evidence from the window) \| `transport_gap` (our browser queue overflowed) \| `blind_spot` (a boundary in the page, such as a cross-origin frame or a closed shadow root) \| `other`. Three owners, three fixes, and one bar on a dashboard until this existed. ONE value, not a list: a multi-value property is not something a breakdown can group by, so the first is sent, ours before the page's. **Absent whenever the capture was clean**, so its presence is itself the signal. Reported as `other` rather than omitted when the block says dirty and names nothing: a gap there would read as "no unclean verdicts happened". |
|
|
145
|
+
| `bug.attribution` | `bug_found` | `app` \| `request` \| `reticle`: whose fault the defect was. **Absent means unclassified**, never `app`. See below. |
|
|
140
146
|
| `outage.stage` / `outage.reason` / `outage.attempts` | `mcp_connection_lost` | which stage of the outage, why the stream went away (closed `OutageReason`, `other` for anything unnamed), and how many reconnects had been tried. See below. |
|
|
147
|
+
| `installSource` | `reticle_installed`, `init_completed` | WHICH published route brought this install in, from core's closed `InstallSource`. Read from one self-declared marker (`RETICLE_INSTALL_SOURCE`) and NEVER inferred, so `unknown` is expected to dominate until every channel's own copy of the install command carries it. See below. |
|
|
148
|
+
| `init.confirmation` | `init_completed` | what `init` SAW after writing, from core's closed `InitConfirmation`: `connected` (an app carrying the SDK reached the daemon while it watched, and it is the only value that means installed) \| `no_daemon` (nothing was listening, so no session could arrive) \| `no_session` (a daemon was up and no app connected inside the window). **Absent means it never looked**, which is every scripted run: `init` waits only when a human is at the terminal. Read absent as "not measured", never as a failure to connect. |
|
|
149
|
+
| `automation` | every event | ADVISORY hint that the run looks automated when `CI` does not say so, from core's closed `AutomationHint`: `container` \| `hosted_workspace` \| `no_tty`. `ci` reads one environment variable set only by a runner, so a gate driven from a cloud sandbox lands as a human at a machine. **Never a filter**: people work in containers, in Codespaces, and over ssh with no terminal, and dropping a row because this is set drops real users. Absent means nothing looked automated, not that a human was present. |
|
|
141
150
|
| `tzOffsetMin` | every event | minutes offset from UTC. One integer, no location. |
|
|
142
151
|
| `versionChange.nudged` | `version_changed` | an agent had been told about exactly this version recently, so the nudge plausibly caused the update. The daemon that nudges and the `reticle update` that acts are different processes, so a marker file joins them. |
|
|
143
152
|
|
|
144
|
-
## Why
|
|
153
|
+
## Why they stopped: `tool_refused`
|
|
154
|
+
|
|
155
|
+
The refusal path computes a precise diagnosis, hands it to the agent as prose, and throws it away. So the biggest cohort in the funnel, the users who attach an agent and never drive, emitted nothing at all and was reachable only by subtracting two other numbers. Half of issue #172.
|
|
156
|
+
|
|
157
|
+
- `refusal_tool`: which tool, from our own fixed namespace. Never app data.
|
|
158
|
+
- `refusal_reason`: the closed `RefusalReason`: `no_session` | `no_match` | `unsupported` | `bad_args` | `not_ready` | `other`. Four different owners, and one undifferentiated "they stopped" number is actionable by none of them.
|
|
159
|
+
- `refusal_retried`: the call immediately before this one was the same tool, also refused.
|
|
160
|
+
|
|
161
|
+
Two things about it are deliberate and easy to get wrong later.
|
|
162
|
+
|
|
163
|
+
**The reason is derived from the recovery table, not from a second list of patterns.** `error-recovery.ts` is the one place a thrown message becomes a next action; `REASON_OF` is a `Record` over its keys, so a recovery added without a reason does not compile. A parallel regex list would have been correct the day it was written and silently wrong at the next addition. That is rule 4, on the exact kind of value that cannot be recovered afterwards.
|
|
164
|
+
|
|
165
|
+
**`retried` lands on the RETRY, not on the first refusal.** Reporting it the other way round means holding the first event back until the next call reveals whether one came, which loses it entirely for an agent that gives up, and that agent is the whole population this event exists to describe. So a refusal is sent the moment it happens, and the retry that follows carries the flag. Count `retried: true` for retries; the ratio against all refusals is whether our diagnosis gets anybody unstuck.
|
|
166
|
+
|
|
167
|
+
Capped at 50 per daemon run. Volume is part of this taxonomy's design and a stuck agent is exactly the shape that produces hundreds; `consecutiveRepeats` on the session summary still reports how long the loop ran.
|
|
168
|
+
|
|
169
|
+
## Which route brought them in: `installSource`
|
|
170
|
+
|
|
171
|
+
Four install routes ship at once (the SKILL.md paste URL, an `npx skills add` package, a Claude Code plugin, and docs.reticle.sh), and not one install could be attributed to any of them. Every decision about where to spend distribution effort was made blind.
|
|
172
|
+
|
|
173
|
+
There is exactly ONE mechanism, and it is a declaration rather than a detection: the channel sets `RETICLE_INSTALL_SOURCE` on the process that runs the install, and `install-source.ts` narrows it against core's `InstallSource`. Anything unrecognised reports `unknown`; an echo would put whatever somebody exported onto the wire.
|
|
174
|
+
|
|
175
|
+
Nothing infers. Three things look like signals and are not: `npm_config_user_agent` says npm ran us, and all four routes go through npx; a `.claude-plugin/` directory or an installed skill folder says a route is PRESENT, not that it ran `init`, and both are present on any machine that tried more than one; and which command ran first says nothing about who told the user to run it.
|
|
176
|
+
|
|
177
|
+
So `plugin` is the only route detectable without anybody typing anything (the plugin registers the MCP server and sets the marker in its `env`). `skill_file`, `npx_skill`, `docs_site` and `readme` are detectable only where that channel's own published copy of the install command carries the marker, and each of those is a separately published artifact. `cli_direct` is not detectable at all.
|
|
178
|
+
|
|
179
|
+
**`unknown` is therefore expected to be the largest bucket, and shrinking it is a distribution job rather than a classifier job.** Read a small `unknown` as a marker that spread. A guessed attribution would be worse than none: it is the number distribution decisions get steered on, and once a guess sits in the same column as a measurement the two cannot be told apart.
|
|
180
|
+
|
|
181
|
+
## Why a verdict came out that way: `verification.reason`
|
|
145
182
|
|
|
146
183
|
`verified` has three values. The rule that produces it has **eleven clauses**. Everything in between was thrown away at the moment it was known.
|
|
147
184
|
|
|
148
|
-
Captured against the real classifier: `verified: 'unknown'` covered "the agent malformed the call", "the consequence was already true", "the app answered 202", "a 2xx body went unread", "the capture was not clean", "nothing was asserted at a real grade" and "the page never settled"
|
|
185
|
+
Captured against the real classifier: `verified: 'unknown'` covered "the agent malformed the call", "the consequence was already true", "the app answered 202", "a 2xx body went unread", "the capture was not clean", "nothing was asserted at a real grade" and "the page never settled": **seven causes, two wire payloads**. They belong to three different owners (the agent, the app, Reticle) and need opposite responses: teach the agent, wait and re-check, or ship a fix. On a dashboard they were one bar. `verified: 'no'` collapsed the same way: "channels disagree" (Reticle earning its keep) and "the agent's predicate failed" were the same string.
|
|
149
186
|
|
|
150
|
-
`VerifiedReason` lives in `@reticlehq/core` and is the **single list**. `decideVerified` returns a member from every clause, so a new clause cannot compile without one; `verified.test.ts` drives all ten and fails if a member exists that no clause produces. `verification-of.ts` narrows the result field against `Object.values(VerifiedReason)
|
|
187
|
+
`VerifiedReason` lives in `@reticlehq/core` and is the **single list**. `decideVerified` returns a member from every clause, so a new clause cannot compile without one; `verified.test.ts` drives all ten and fails if a member exists that no clause produces. `verification-of.ts` narrows the result field against `Object.values(VerifiedReason)`, and anything else is dropped rather than forwarded, because a string nobody can group by is worse than a gap. **Nothing re-lists these**, including the battery spec, which imports the enum from core's build.
|
|
151
188
|
|
|
152
189
|
Optional on purpose: a suite verdict (`flow_verify`) is a pass/fail with no clause behind it, and an older sender has none. Absent means unclassified.
|
|
153
190
|
|
|
154
|
-
## Whose defect it was
|
|
191
|
+
## Whose defect it was: `bug.attribution`
|
|
192
|
+
|
|
193
|
+
It shipped twice and was wrong both times. A real drive found that across two full runs EVERY `attribution: 'app'` was a misattribution, while the one defect that genuinely was a bad agent predicate carried none. A single session would have published "2 defects in the app" against a true count of 0. So it was removed, and then absence made "nobody classified this" and "we looked and could not tell" the same fact, which is the other half of the same problem.
|
|
194
|
+
|
|
195
|
+
It is back with two rules, and both are the thing the earlier versions lacked. Issue #122.
|
|
196
|
+
|
|
197
|
+
**Always present.** `unclassified` is a value, not a gap. Absence would mean an old sender or a path that forgot; a value means the classifier ran and declined. Count `app` for defects in anybody's product; exclude `unclassified` rather than folding it in.
|
|
155
198
|
|
|
156
|
-
`
|
|
199
|
+
**`app` requires positive evidence.** Something the app itself did: a request that came back failed, a signal the app fired carrying data that disagrees with its own screen, a written field echoed back changed. Never "nothing else explained it". That line already exists in core, as `ABSENCE_DERIVED_CONTRADICTIONS`, the kinds inferred from something NOT having happened inside a window Reticle chose the end of, and it is the same line that decides whether a verdict may say `no`. Reusing it rather than inventing a second judgement beside it is the point: every historical misattribution was an absence-derived kind, so this rule produces zero of them on the data that broke the last two versions.
|
|
157
200
|
|
|
158
|
-
|
|
201
|
+
Everything else is `unclassified` on purpose. A failed `element.present` covers "the button is missing", "the API is down" and "the agent mistyped a testid" identically. A console error can be the app, a browser extension, or a framework's dev overlay, and one of those was a real false positive here. A replay regression says a flow that used to pass no longer does, which is a regression somewhere, and the row does not say whether the app changed or a selector strategy of ours did.
|
|
159
202
|
|
|
160
|
-
|
|
203
|
+
Driver-side causes need no bucket of their own: a stale ref, a malformed call and a lost session produce no `bug_found` at all, because those paths are excluded upstream in `bugsInResult` before anything is counted.
|
|
161
204
|
|
|
162
|
-
## What is NOT a crash
|
|
205
|
+
## What is NOT a crash: expected disconnects
|
|
163
206
|
|
|
164
|
-
`runtime_crashed` answers exactly one question: is Reticle stable. One real session put **nine** events into it, all `write EPIPE
|
|
207
|
+
`runtime_crashed` answers exactly one question: is Reticle stable. One real session put **nine** events into it, all `write EPIPE`: the MCP client closed its half of the stdio pipe and the next `process.stdout.write` failed, which is how a client is supposed to leave.
|
|
165
208
|
|
|
166
209
|
`daemon-resilience.ts` matches `err.code` against `EPIPE` / `ECONNRESET` / `ERR_STREAM_DESTROYED` and logs `reticle_daemon_client_disconnected` (or `reticle_mcp_proxy_client_disconnected`) instead of emitting. Two rules make this safe rather than a hole:
|
|
167
210
|
|
|
@@ -170,19 +213,19 @@ It comes back when a verdict carries the reason it came out that way. Today the
|
|
|
170
213
|
|
|
171
214
|
A disconnect is also **no longer fatal for the daemon**: Node's "process state is undefined" guidance is about a throw that escaped everything, not about writing to a socket somebody closed, and exiting there let one departing client take down the daemon serving every other agent.
|
|
172
215
|
|
|
173
|
-
## Did MCP stay up
|
|
216
|
+
## Did MCP stay up: the `outage` block
|
|
174
217
|
|
|
175
|
-
The transport-stability metric shipped with an **empty payload** for months, and it is the exact failure this page opens with. `reportMcpOutage` passed `{ outage: { stage, reason, attempts } }`, `TelemetryExtra` declared the field, and it typechecked
|
|
218
|
+
The transport-stability metric shipped with an **empty payload** for months, and it is the exact failure this page opens with. `reportMcpOutage` passed `{ outage: { stage, reason, attempts } }`, `TelemetryExtra` declared the field, and it typechecked. But `emit()` builds its event from an **explicit allow-list of keys** and `outage` was not on it, nor in the `blocks` flattening map, nor in core's `TelemetryEventSchema`. Two deliberately different outages produced byte-identical events. Nothing threw, no test went red, and the data for that whole period cannot be recovered.
|
|
176
219
|
|
|
177
220
|
The lesson is not "wire the field". It is that **the battery asserted the event ARRIVED and never that it carried anything**, and a kind-only assertion cannot see an empty payload. When you add an event kind, the live check has to assert the FIELDS.
|
|
178
221
|
|
|
179
|
-
- `stage`
|
|
180
|
-
- `reason`
|
|
181
|
-
- `attempts
|
|
222
|
+
- `stage` is `first` (this session lost MCP at all) or `budget_spent` (it stopped retrying). These are the two facts the event exists to separate: the share of sessions that lose MCP, and the share where it never came back on its own.
|
|
223
|
+
- `reason` is the closed `OutageReason`: `sse_ended` | `sse_error` | `sse_aborted` | `sse_closed` | `connect_error` | `other`. The proxy's own reason strings are free text that also feeds a log, so `mcp-outage.ts` narrows them and reports **`other`** for anything unnamed. A classifier that cannot say "I don't know" lies instead, and an unbounded string must never reach the wire.
|
|
224
|
+
- `attempts`: consecutive reconnects tried when this was reported.
|
|
182
225
|
|
|
183
226
|
Still true and worth knowing when you query it: `mcp_connection_lost` carries **no `sessionId`** (it fires from the proxy process, not the daemon), and is capped at two per proxy process by design.
|
|
184
227
|
|
|
185
|
-
## Recording locally instead of sending
|
|
228
|
+
## Recording locally instead of sending: `RETICLE_TELEMETRY_FILE`
|
|
186
229
|
|
|
187
230
|
Set it to a path and every event is appended there as one JSON object per line, and **nothing is sent**. The payload is the one the wire would have carried, built by the same code and redacted by the same rules, so what a run records is what a user would have sent.
|
|
188
231
|
|
|
@@ -191,7 +234,7 @@ It exists for two reasons that pull the same way:
|
|
|
191
234
|
- **A release sweep is not a user.** Driving dozens of sessions through a gate emits real `daemon_started` / `verification_completed` / `bug_found` events, indistinguishable in PostHog from people. Test runs polluting the numbers is the same class of error as counting `cli_command_run { mcp }` as human intent: the metric stops describing what it claims to.
|
|
192
235
|
- **Verifying telemetry should not need a hand-rolled HTTP server.** Ad-hoc harnesses are how a check ends up measuring nothing.
|
|
193
236
|
|
|
194
|
-
One deliberate exception to the rules above: `RETICLE_TELEMETRY_FILE` keeps telemetry ENABLED inside a Reticle source checkout. The checkout guard exists to stop us phoning home, and writing a local file is not phoning home
|
|
237
|
+
One deliberate exception to the rules above: `RETICLE_TELEMETRY_FILE` keeps telemetry ENABLED inside a Reticle source checkout. The checkout guard exists to stop us phoning home, and writing a local file is not phoning home, while a release sweep is driven from exactly there, so a sink that inherited the guard would record nothing and look like it had worked.
|
|
195
238
|
|
|
196
239
|
`sent: true` from `reticle_feedback` means the record landed in the file, which is the honest reading of "captured" for a recorded run. An unwritable path degrades to a no-op and reports `false`; it never takes the daemon down.
|
|
197
240
|
|
|
@@ -200,13 +243,13 @@ One deliberate exception to the rules above: `RETICLE_TELEMETRY_FILE` keeps tele
|
|
|
200
243
|
| You are adding | Do this | Enforced by |
|
|
201
244
|
| --- | --- | --- |
|
|
202
245
|
| **A tool** | Add it to `TOOLS`. Nothing else. If its name implies a verdict (`assert`/`verify`), also add it to `VERIFICATION_TOOLS` | `telemetry-contract.test.ts` |
|
|
203
|
-
| **A verdict-producing tool** | `VERIFICATION_TOOLS
|
|
246
|
+
| **A verdict-producing tool** | Add it to `VERIFICATION_TOOLS`. Otherwise it emits no `verification_completed` and stops counting toward the product's headline metric | ✓ |
|
|
204
247
|
| **A contradiction / anomaly kind** | Add it to core's enum only. `bug-found.ts` derives from it | ✓ |
|
|
205
248
|
| **A new finding shape** in a tool result | Teach `bugsInResult` the field. Add a case to the contract test | ✓ |
|
|
206
|
-
| **A failure path** (connect, install, crash) | Classify it into an enum with an explicit `OTHER` bucket
|
|
207
|
-
| **An event kind** | Add to `TelemetryEventKind` + a payload schema + emit it + add a live check to `apps/e2e/specs/telemetry-events-test.mjs` that asserts the **fields**, not just that it arrived | partly
|
|
208
|
-
| **A block on `TelemetryExtra`** | Also add it to the `emit()` event build, the `blocks` flattening map, AND `TelemetryEventSchema`. Missing any one of the three drops the payload in silence
|
|
209
|
-
| **A dispatch path** that bypasses `runTool` | Give it a reporter like `run-telemetry.ts`, or it is invisible | ✗ **not enforced
|
|
249
|
+
| **A failure path** (connect, install, crash) | Classify it into an enum with an explicit `OTHER` bucket; a classifier that cannot say "I don't know" lies instead | ✓ |
|
|
250
|
+
| **An event kind** | Add to `TelemetryEventKind` + a payload schema + emit it + add a live check to `apps/e2e/specs/telemetry-events-test.mjs` that asserts the **fields**, not just that it arrived | partly; the live check is on you |
|
|
251
|
+
| **A block on `TelemetryExtra`** | Also add it to the `emit()` event build, the `blocks` flattening map, AND `TelemetryEventSchema`. Missing any one of the three drops the payload in silence (see `outage`) | ✗ **not enforced; be careful** |
|
|
252
|
+
| **A dispatch path** that bypasses `runTool` | Give it a reporter like `run-telemetry.ts`, or it is invisible | ✗ **not enforced; be careful** |
|
|
210
253
|
|
|
211
254
|
## Verifying it actually works
|
|
212
255
|
|
|
@@ -217,10 +260,10 @@ pnpm test:unit # the contract test + the finger
|
|
|
217
260
|
node apps/e2e/specs/telemetry-events-test.mjs # fires every event at a real endpoint, checks it lands
|
|
218
261
|
```
|
|
219
262
|
|
|
220
|
-
The second is the one that matters. It drives the real built modules against a real capture server
|
|
263
|
+
The second is the one that matters. It drives the real built modules against a real capture server (real network, real process semantics, real redaction) and asserts each event **arrives**. Half its checks are leak checks, asserting that secrets, passwords, customer emails and home directories are _absent_.
|
|
221
264
|
|
|
222
265
|
**Both halves are mutation-tested.** Reintroducing the fire-and-forget bug fails 9 checks; disabling redaction fails 3. A guard that cannot fail is theatre, so these are periodically proven to bite.
|
|
223
266
|
|
|
224
267
|
## The privacy line, in one sentence
|
|
225
268
|
|
|
226
|
-
We measure **that** something happened and **what class** of thing it was
|
|
269
|
+
We measure **that** something happened and **what class** of thing it was, never **what** it was, in whose app, or containing what.
|