@andorra-labs/code 0.0.0-stage → 0.1.4

package/tests/product-telemetry-test-plan.md

@@ -0,0 +1,536 @@
+# Product telemetry QA
+
+## Scope and prerequisites
+
+This plan covers ADR 0021's collector: anonymous, plan-gated product telemetry. It reports how the
+product is used — lifecycle, reliability, model and tool aggregates, workflow shape and a bounded
+workspace shape — and never what it is used on. The preference and the plan gate are covered by
+[`analytics-settings-test-plan.md`](analytics-settings-test-plan.md).
+
+What is in scope:
+
+- The closed schema (`src/telemetry/schema.ts` and its Python mirror `infra/andorra-auth/telemetry_schema.py`).
+- The emitter, the per-engine outbox, and the drainer that posts to the ingest endpoint.
+- The run lifecycle: one `run.started` and one `run.finished` per run, a `session.started` per
+  created conversation, and an `interrupted` finish for a run recovery found dead.
+- The hosted ingest stack in `infra/andorra-auth/template.json`: the dedicated REST API with WAF,
+  the ingest Lambda, the SQS queue and DLQ, the writer, the S3 bucket, the rollup job and the
+  alarms.
+
+Prerequisites:
+
+- Node 22.19+ and a built worktree: `npm ci`, `npm run build`.
+- Python 3.9+ for the handler tests: `python3 -m unittest discover -s infra/andorra-auth -p 'test_*.py' -v`.
+- A disposable home for every run, including on failure:
+
+  ```bash
+  home=$(mktemp -d) || exit 1
+  export ANDORRACODE_HOME="$home"
+  # ... run ...
+  rm -rf "$ANDORRACODE_HOME"
+  ```
+
+- A local ingest endpoint, because nothing is deployed by this change and the drainer is a no-op
+  while `ANDORRA_TELEMETRY_URL` is unset. A tiny capture server is enough:
+
+  ```bash
+  node -e "require('http').createServer((q,s)=>{let b='';q.on('data',c=>b+=c);q.on('end',()=>{console.log(b);s.writeHead(202,{'content-type':'application/json'});s.end('{}')})}).listen(0,'127.0.0.1',function(){console.error(this.address().port)})"
+  ```
+
+  Export the printed port: `export ANDORRA_TELEMETRY_URL="http://127.0.0.1:<port>"`.
+- A provider that answers without network is not required for the schema and outbox cases; the
+  lifecycle cases need a run, which needs a configured provider or the scripted mock provider used
+  by `test/telemetry-emitter.test.ts`.
+
+## How to use the feature
+
+Nothing needs to be enabled. Telemetry is on for every plan except a confirmed Paid account that
+turned it off; there is no prompt and no first-run screen.
+
+1. Point the installation at the capture server and give it a fresh home.
+2. Start a run, for example `node dist/src/cli.js --prompt "Say hello"`, and let it finish.
+3. The CLI drains its spool before exit. The capture server prints one `run.started` and one
+   `run.finished` body, joined by the same `runId`.
+4. Inspect the bodies: every key is one the ADR lists, every value is a number, boolean, timestamp
+   or product-owned enumeration, and `installId` is 32 hex characters.
+5. The outbox is under `$ANDORRACODE_HOME/telemetry/outbox/`; after a successful drain the spool
+   file is gone. `$ANDORRACODE_HOME/telemetry/install-id` holds the random install id that every
+   event carries, and `state.json` appears once an account plan has been confirmed; it holds only
+   subject hashes, tiers, preferences and timestamps.
+
+To reset: delete the disposable home. There is nothing else to clean up, and no listener outlives
+the command that started it.
+
+## Test cases and expected results
+
+### Schema and emitter
+
+- **TE-01** A valid `run.started` and `run.finished` serialize with exactly the ADR's key sets.
+  Any extra key (a `prompt`, a `model`, a path) is absent from the serialized body.
+- **TE-02** An unknown key is dropped by `validateTelemetryEvent`, not stored.
+- **TE-03** A known field of the wrong type, outside its enum, or outside its bound rejects the
+  whole event; no partial record is stored.
+- **TE-04** `retries > attempts`, `toolFailures > toolCalls`, `tools[].failures > tools[].calls`, a
+  `tools[]` sum that disagrees with `toolCalls`, a duplicate model tuple, and a duplicate tool name
+  each reject the event.
+- **TE-05** A `run.started` carrying `toolCalls`, `outcome` or `models` is rejected: those belong to
+  the finish.
+- **TE-06** A `session.started` carrying `runId` is rejected.
+- **TE-07** A `ts` older than 30 days or more than 5 minutes ahead is rejected; a timestamp is
+  minute precision.
+- **TE-08** A model id not on the allowlist is reported as `other`; a local base URL reports
+  `endpoint: "local"`; an MCP tool name reports `mcp`; `update_todos` is not reported at all.
+- **TE-09** A model tuple in which only some requests reported input tokens has no `inputTokens`
+  field. A mixed-provenance overflow record reports `coverage.costMicroUsd: "partial"` and no
+  summed cost.
+- **TE-10** A failure maps totally: `succeeded`→`completed`; `exhausted` with `wallTimeMs`→
+  `timeout`; `repeatedToolFailure`→`tool_error`; `loop.maxIterations:<nodeId>` and any unknown
+  reason→`limit`; a provider failure→`provider_error`; a classified overflow→`limit`.
+- **TE-11** A task category follows the ordered rules: no edit and no test→`investigate`; only
+  documentation edits→`docs`; only test edits→`test`; a created file→`implement`; otherwise→`fix`.
+- **TE-12** The language and framework mix is at most five distinct values, ranked deterministically,
+  and never contains a path.
+- **TE-13** A run emits exactly one start and one finish, both carrying the same random `runId`; a
+  resumed attempt is a new run with a new id.
+- **TE-14** A `session.started` is emitted when a conversation is created and not when one is
+  reopened.
+- **TE-15** A process killed mid-run leaves its open record; the next start marks it `interrupted`
+  and enqueues one `interrupted` finish using the `runId` from that record.
+- **TE-28** `appVersion` is the built manifest's version, never `unknown`; a lookup that resolves
+  one directory short (inside `dist/`) reports `unknown` and silently drops every event out of
+  version comparisons.
+
+### Plan gate and outbox
+
+- **TE-16** A Free or never-confirmed installation reports even with `enabled: false`.
+- **TE-17** A confirmed Paid installation with `enabled: false` sends nothing; the drainer discards
+  what was queued.
+- **TE-18** Turning reporting off increments the preference generation and purges the outbox; an
+  entry appended concurrently under the old generation is discarded by the next drain.
+- **TE-19** The outbox is bounded by whole runs: eviction never leaves a finish without its start.
+  Anything older than 30 days is discarded.
+- **TE-20** A spool whose owner died is adopted by the next coordinator through its lock; a live
+  engine's spool is left alone.
+- **TE-21** A `409` from the ingest counts as success (the object already exists). A `5xx`, a
+  network failure, and a `403` keep the record for the next drain — the WAF's rate rule answers
+  `403` on this unauthenticated route, so it is a throttle, not a rejection. Any other `4xx` drops
+  it rather than poisoning the queue.
+- **TE-22** No endpoint configured means nothing is uploaded and nothing is lost.
+
+### Ingest stack
+
+- **TE-23** An unauthenticated `POST /events` with a valid record answers 202 and enqueues one SQS
+  message; `GET` answers 405.
+- **TE-24** A body over 64 KiB answers 413 before parsing. An unimplemented `schema` version,
+  an invalid enum, a broken counter relationship, or a breakdown that does not sum to its total
+  answers 400 and stores nothing.
+- **TE-25** The writer stores one object per event at `{yyyy}/{mm}/{dd}/{eventId}.json` with
+  `If-None-Match: *`; the date comes from the event, so a retry across midnight addresses the same
+  key. The object is written **before** the partition generation is incremented, so a rollup that
+  sees the new generation also sees the object. A 412 is success and still increments (the first
+  attempt may have died before marking the partition); a failed write increments nothing.
+- **TE-26** The rollup recomputes and replaces a changed partition; a partition whose generation
+  equals the stored one is skipped without listing; a generation that moves during the listing
+  causes a retry; a transient object read raises rather than storing an incomplete partition.
+- **TE-27** The template defines the dedicated REST API with a WAF rate-based block rule, the
+  ingest/writer/rollup Lambdas, the SQS queue with a dead-letter queue and 4-day retention, the
+  DLQ-depth and source-depth/age alarms, the S3 bucket with versioning disabled and a 13-month
+  expiry, and the rollup table with a 25-month TTL. The event source mapping carries no
+  `RedrivePolicy`; redrive belongs to the queue.
+- **TE-29** A successful `/me` records the named tier for that subject, so a Paid opt-out can
+  exist, a Free plan is refused the switch, and a second signed-in account is `unknown` until its
+  own answer arrives.
+- **TE-30** A CLI run whose graph preflight fails still sends a joined pair: the start is emitted
+  before the finish that follows it, and the finish is not sent without its start.
+- **TE-31** A parallel graph branch's telemetry is merged once, in the group settlement; the later
+  activation accounting skips it, so attempts, tokens and tool calls are not doubled.
+- **TE-32** An argumentless `/feedback` obeys the plan gate exactly like `/feedback off`, and a
+  flip to off purges the outbox.
+- **TE-33** A process whose startup snapshot says reporting is on still honors an opt-out another
+  process saved: the preference and generation are re-read from disk for each emit and each drain.
+- **TE-34** Opting out aborts this process's in-flight upload as well as removing queued entries, so
+  a request cannot store a pre-opt-out event after the choice was saved.
+- **TE-35** A recovered interrupted finish gates on the `accountSubject` written into its own open
+  record, not on whichever account most recently confirmed a plan on a shared installation.
+- **TE-36** A preference change updates the cached entry's `preference`, so a Paid opt-out is
+  protected from eviction by later confirmations from other accounts.
+- **TE-37** A successful `/me` that names no tier records `unknown`, replacing a stale Paid
+  entitlement; an outage or a signed-out answer records nothing and leaves the entry alone.
+- **TE-38** The template orders the stage after the API Gateway account, attaches the web ACL after
+  the stage exists, and creates the telemetry log groups at the `/aws/lambda/` paths the functions
+  write to.
+- **TE-39** `pricingVersion` is validated against the server's shipped set (empty at launch), so an
+  invented version label is refused.
+- **TE-40** A browser-owned conversation emits its own joined lifecycle pair and drains at the end,
+  so `/browser` runs appear in the same counts as CLI and desktop runs.
+- **TE-41** The edge blocks a request whose `Content-Length` exceeds 64 KiB before it reaches the
+  integration, and the dead-letter alarm fires on the first waiting message (threshold `0`).
+- **TE-42** The writer stores an event that waited in the queue past the 30-day window: freshness is
+  judged once, at receipt, not again at consumption.
+- **TE-43** The ingest refuses an overflow record that is not last or that names a real provider,
+  endpoint, model or role; the client validator applies the same rule.
+- **TE-44** A rollup rechecks the partition generation after folding, so a scan that started before
+  a writer landed cannot overwrite the newer, complete rollup.
+- **TE-45** A chained shell line (`cd packages/api && npm test`) is recognized as a test command,
+  and every returned config snapshot — graph selection and permission changes included — carries
+  the plan gate, so the Analytics page never loses its switch.
+- **TE-46** `attempts` counts every node execution: an action-only graph reports its activations,
+  and a revisited node counts as a retry, while an agent node's own attempt loop is counted once.
+- **TE-47** A browser task that fails while loading or checking its graph still sends a joined pair;
+  a subagent start is retained so a setup failure's finish joins it.
+- **TE-48** A rollup write is conditional on the generation, and a measurement no model reported is
+  omitted from the day's counts instead of published as zero.
+- **TE-49** The QA tools (`qa.test`, `qa.browser`, …) are reported by name rather than folded into
+  `other`, in both the client and ingest enums.
+- **TE-50** The installation-wide outbox bound holds across every spool file, including while no
+  endpoint is configured; the workspace sample is deterministic for a large repository.
+- **TE-51** A browser-owned run reports the signed-in subject and the graph it actually resolved,
+  including when the requested graph id fell back to a default.
+- **TE-52** Switching analytics off purges the outbox; a run that starts while it is already off
+  does not purge again, and a later `on` keeps reporting. The purge is tied to the transition, not
+  to the current value.
+- **TE-53** Two writers recovering the same interrupted run (coordinator and a late finisher) do
+  not interleave: the recovery rewrite takes a per-run save lock and removes the lock directory,
+  so the result is one record with one interrupted finish.
+- **TE-54** The ingest's per-installation rate limit refuses the 121st event in a minute with 429
+  and a `Retry-After`, while other installations are unaffected; the accepted events are still
+  enqueued.
+- **TE-55** A rollup over a day far larger than one Lambda's sequential read budget folds with
+  bounded parallel object reads and produces the same counts as a sequential fold.
+- **TE-56** A workspace whose framework is declared in a nested package manifest (for example
+  `packages/api/pyproject.toml` under a root `package.json`) reports that framework; a root
+  manifest no longer hides it, and the manifest set is bounded.
+- **TE-57** A rollup treats an object that is merely older than the receipt window as valid data
+  (freshness is not re-enforced after the queue delay), so a backlog does not silently drop events.
+- **TE-58** An outbox drain with no configured endpoint still applies the installation-wide bound,
+  so an offline installation cannot grow past the event and byte caps.
+- **TE-59** A drain that started while reporting was on re-reads the preference between posts:
+  after another engine opts out, it stops mid-spool and leaves the rest for the purge instead of
+  sending the whole snapshot.
+- **TE-60** Two coordinators that read the same open record before either rewrite lands report the
+  interrupted run once: the record is re-read under the recovery lock and only the reader that
+  still sees `running` rewrites it.
+- **TE-61** A JEV-enabled run's decision requests, completions and applications reach the run's
+  capture even though the decision manager publishes through its own event sink; the reported
+  counters are not zero.
+- **TE-62** A subagent start, finish and drain all name the account the run started under, and the
+  finish repeats the start's `jevEnabled` and routing dimensions even if the configuration changed
+  while the run was in flight.
+- **TE-63** The desktop Analytics page re-reads the plan gate when the account store refreshes —
+  an upgrade picked up on window focus reveals the switch without remounting the section.
+- **TE-64** CLI `session.started` events bind to the signed-in account, so a shared installation's
+  opt-out is not overridden by another account's newer plan-cache entry; the enqueue completes
+  before the CLI exits.
+- **TE-65** A crashed app-server run's open record carries the effective permission mode, so its
+  recovered `interrupted` finish reports the same permission as its start.
+- **TE-66** A tool call interrupted by the user's pause or stop is not counted as a tool failure,
+  in either the run total or the per-tool row.
+- **TE-67** Graph-health preflight probes are reported as their own model tuple: their requests and
+  tokens appear in `models[]`, and a run that failed only the health check still reports a model.
+- **TE-68** `session.started` is emitted for a session whose first turn lands, not for a reopen
+  whose trajectory file is missing because recording degraded.
+- **TE-69** A run that only called a model and executed no tool omits `taskCategory` instead of
+  reporting `investigate`.
+- **TE-70** A modifier node's loop iterations and parallel copies all contribute their captures:
+  models, tools, JEV and task evidence from every copy are reported, not just the last one.
+- **TE-71** A delegated subagent's capture is folded into the parent run, so its model requests,
+  tools and JEV activity appear once in the run's totals under the subagent role.
+- **TE-72** A subagent whose session record cannot be saved finishes under the account and
+  dimensions its start used, not whichever account the plan cache confirmed most recently.
+- **TE-73** An interactive CLI opened and exited without a task emits no `session.started`; the
+  event appears when the first task creates the saved conversation, and a fork still reports one.
+- **TE-74** A daily rollup with mixed measurement coverage keeps the totals the reporting records
+  measured and publishes per-measurement coverage alongside them, omitting only a measurement
+  nobody reported.
+- **TE-75** The `subagent` entry point reports the interrupted finishes it recovers, so a crash
+  followed by another subagent invocation still closes the start.
+- **TE-76** A run that fails in graph loading or model preflight reports zero attempts rather than
+  one it never made.
+- **TE-77** An HTTP 408 from the endpoint keeps the event queued for retry instead of dropping it.
+- **TE-78** An action node's shell call appears in the run's tool totals, its test actions are
+  evidence for `taskCategory`, and an interrupted action is not a tool failure.
+- **TE-79** A repeated `/me` confirmation for the active account refreshes its recency even when
+  its tier and preference are unchanged, so signing out selects that account's entry rather than
+  another account's older one.
+- **TE-80** A crashed CLI run's open record carries the permission mode it started under, so its
+  recovered finish agrees with the start.
+- **TE-81** Action-node accounting attributes a real shell failure to the tool and leaves a
+  user-interrupted call out of the failure count.
+- **TE-82** A health probe that reported no usage produces a tuple with `requests` only — no
+  measured-zero tokens — and a probe that failed after dispatch still leaves correct coverage.
+- **TE-83** More than sixteen model tuples, including a real all-`other` tuple, produce unique
+  identities: the real all-`other` tuple folds into the overflow row and the event is accepted.
+- **TE-84** A rollup does not publish a partition whose listing changed while it was folding (the
+  event-before-generation-bump window), and folds it on a later scan instead.
+- **TE-85** Outbox eviction removes a run whole across spool files: a crashed run's start and the
+  recovered finish in another spool go together, never one without the other.
+- **TE-86** A failed test command still counts as test evidence for `taskCategory`; only a
+  user-interrupted command is excluded.
+- **TE-87** A recovered run whose open record says its start was never queued emits the start
+  before the finish, so no finish is orphaned; a record whose start was queued contributes only
+  the finish.
+- **TE-88** A browser task's root-model requests are reported against the selected provider and
+  endpoint, not bucketed as `other`/`other` because the preseeded model entry lacked them.
+- **TE-89** A browser task's health probes are reported as model tuples, and a task that failed
+  the health check still reports its probe requests.
+- **TE-90** A failed provider request is reported as a model tuple with `requests` only: its
+  attempts are billed, and no token measurement is claimed.
+- **TE-91** A subagent execution is an execution boundary, not a new conversation: it emits its
+  run lifecycle events and no `session.started`.
+- **TE-92** The telemetry local-CI group selects for changes to every capture integration point
+  (`runtime`, `agent-execution`, `agent-modifiers`, `graph-runner`, `graph-health`,
+  `browser-runner`, `membership`).
+- **TE-93** A conversation created through the browser's session endpoint emits one
+  `session.started`, under the signed-in account and its workspace, before the create response.
+- **TE-94** An action node's failed test command is recorded as test evidence before the failure
+  propagates, so the run is not classified `investigate`.
+- **TE-95** A shadow decision request still in flight when its node ends is awaited at
+  finalization, so the finish reports the terminal decision events instead of only the request.
+- **TE-96** A rollup defers a partition whose writer has raised the in-flight counter and folds
+  it on a later run once the counter is back to zero; a counter left raised by a dead writer ages
+  out instead of stalling the day.
+- **TE-97** An opt-out cannot be overtaken by an upload in another process: the preference check
+  and the POST happen inside the spool lock the purge queues behind, so an event is either sent
+  before the opt-out finished or not sent.
+- **TE-98** A browser task whose health probe was rejected by the provider is finalized as a
+  provider failure, not an infrastructure one.
+- **TE-99** A tool from a plugin-provided MCP server is reported in the `plugin` bucket rather
+  than `mcp`, including through the generated `mcp_plugin_…` name.
+- **TE-100** The telemetry Web ACL keeps CloudWatch metrics but stores no sampled requests, so no
+  client address or header is retained by WAF logging.
+- **TE-101** An embedded engine with a non-default home records `/me` plan confirmations in that
+  home, so a Paid opt-out there is obeyed rather than read from the process default.
+- **TE-102** Changing an account's preference refreshes its confirmed recency, so a sign-out
+  selects that account rather than another account confirmed later.
+- **TE-103** The writer raises the counter before writing the object and lowers it in the same
+  atomic update that bumps the generation, so a rollup reading the partition item sees either a
+  live upload or a completed one, never a mixture.
+- **TE-104** Lifecycle pairs stay whole when the preference changes mid-run: a run whose start was
+  suppressed emits no finish, and a run whose start went out finishes even if reporting was turned
+  off in between.
+- **TE-105** A rollup folds a large partition in bounded chunks, so memory does not scale with the
+  number of events in the day.
+- **TE-106** A browser task whose workspace cannot be resolved still emits a joined
+  `run.started`/`run.finished` pair, the finish classified `infrastructure`, because the start is
+  queued before the fallible setup.
+- **TE-107** A CLI process killed between writing its open record and queueing the start is
+  recovered as a pair: the record's `startSent` is `false` until the start is actually queued, and
+  the recovered finish synthesizes the missing start.
+- **TE-108** A recovered run reports the permission mode its surface stored under
+  `telemetry.permissions` (CLI) or `telemetry.permissionMode` (app-server), not `unstated`.
+- **TE-109** A subagent's open record carries the account subject and `startSent`, so a crash does
+  not hand its finish to whichever cached account is newest.
+- **TE-110** A remote child that dies mid-run produces the interrupted finish for the start it
+  queued; the reconciliation result is reported rather than discarded.
+- **TE-111** A shadow preflight that settles after its capture was merged still contributes its
+  JEV counters to the run's finish.
+- **TE-112** A health probe whose `usage` omits a token field reports it as absent, so the tuple's
+  coverage does not claim a measured zero.
+- **TE-113** A `/me` confirmation that was in flight when `/feedback off` landed keeps the newer
+  preference while still updating the tier.
+- **TE-114** A recovery that dies after marking a record interrupted but before queueing its finish
+  is picked up by the next pass from the persisted pending flag, under the same event id, and the
+  flag is cleared once the finish is on the spool.
+- **TE-115** A finish the emitter could not queue (a full disk, a busy spool lock) leaves the
+  pending marker in place, so the next pass retries it instead of orphaning the start.
+- **TE-116** A writer that dies between taking its in-flight lease and dropping it delays one
+  rollup rather than every later one: the lease expires on its own and a busy partition keeps
+  publishing.
+- **TE-117** Outbox eviction leaves a half-written run alone while its owning engine is alive, and
+  removes a completed pair whole even when the pair lives in another spool.
+- **TE-118** The app-server stamps `startSent` from the actual start result, so a gate-suppressed
+  start is not claimed by the open record.
+- **TE-119** The workspace scan is bounded by entries examined, so a tree of thousands of
+  directories with few files cannot delay session creation.
+- **TE-120** A model-backed memory distillation is counted once: the run totals and the model
+  tuple agree with the provider's own usage.
+- **TE-121** A post-tool decision check counts each `decision.*` event once, not once per layer.
+- **TE-122** A settings save that repeats an opt-out does not purge the outbox; only the update
+  that transitions on-to-off does, decided from the configuration the lock read.
+- **TE-123** A stored event whose generation bump fails keeps its in-flight lease, so no rollup
+  publishes a partition that omits it.
+- **TE-124** A duplicate delivery releases its lease after the bump, so the day is not deferred
+  for the whole lease lifetime.
+- **TE-125** A browser run's start carries the graph and settings it actually ran with, because it
+  is queued after they are resolved; a setup failure still reports a matched placeholder pair.
+- **TE-126** When more than five frameworks are detected, the reported five are those the most
+  manifests declare.
+- **TE-127** A subagent execution emits run lifecycle events but no `session.started`; only a new
+  user conversation does.
+- **TE-128** A CLI run reports its graph-health probes as their own tuple: the finish's `models[]`
+  carries an auxiliary tuple with the probe's requests, and a run that failed the health check
+  alone still reports a model.
+- **TE-129** Running `/feedback off` when the saved preference is already off does not purge the
+  outbox, so events another process legitimately queued after the original opt-out survive; a real
+  on-to-off transition still purges.
+- **TE-130** Every shipped default provider model is reported by name: the Anthropic default
+  (`claude-sonnet-4-6`) is in both the client allowlist and the server's `MODELS`, so an
+  out-of-the-box run is not bucketed as `other`.
+- **TE-131** A model-backed memory distillation is a billed request in the run's totals, so it
+  reports its own auxiliary model tuple with its requests and tokens; a run whose only model call
+  was the distillation still reports a model.
+- **TE-132** The endpoint stamp refuses a bare `execute-api` host by giving it the `/v1` stage
+  before writing, refuses any other shape, and `--check` fails on an empty or unstaged committed
+  value while passing on a staged one.
+- **TE-133** A release cannot publish an inert drainer: `release.yml` stamps `TELEMETRY_API_URL`
+  into the artifact when the repository variable is set, and otherwise fails on `--check` when the
+  committed endpoint is empty or invalid.
+- **TE-134** A browser server started on a non-default home reports its runs into that home: a run
+  driven through `startBrowserServer({ directory })` writes its `run.started`/`run.finished` pair
+  there, and the process default home receives nothing.
+- **TE-135** A recovery pass that dies after queueing a synthesized start but before its finish
+  cannot store a second start: the synthesized start carries a derived event id that a later pass
+  reuses.
+- **TE-136** A run recovered by a different surface than the one that produced it keeps the
+  producer's label: the open record's `surface` overrides the recovering process's, so the joined
+  pair reads `cli` after a desktop app-server finds it.
+- **TE-137** A capture merged while two shadow preflights are pending transfers their counters
+  once: no preflight's counts are added again by another's settlement, so `completed` cannot
+  exceed `requested` and the finish is not dropped.
+- **TE-138** A paid opt-out that purges a run's start also drops its paired finish: the handle
+  remembers the generation its start was queued under, and a finish from an older generation is
+  not stored, so no unmatched finish can reach another account's drain.
+- **TE-139** A released SemVer is reported by name: the client and the ingest accept a bounded
+  public version rather than a fixed list, so a release carried by the manifest is not stored as
+  `other`, while a private build string still is.
+- **TE-140** A CLI run whose health probe was rejected by the provider is classified
+  `provider_error`, not `infrastructure`, because the failure evidence is carried into the finish
+  the run still emits.
+- **TE-141** A subagent whose health probe was rejected is classified the same way, from the
+  evidence stored on its active run.
+- **TE-142** A first turn that fails during provider lookup or graph resolution still emits
+  `session.started`: the event is queued at the accepted turn, before that setup.
+- **TE-143** A batch item whose upload lease could not be taken is reported as failed and retried;
+  the writer does not store the object without the lease, and a rate-limited installation receives
+  a `Retry-After` for the rest of the current minute.
+- **TE-144** A recovery whose synthesized start cannot be queued does not queue its finish, so a
+  transient failure cannot leave an unmatched finish behind the pending marker.
+- **TE-145** An expired lifecycle run leaves the outbox whole: a run whose start aged out across
+  the window does not upload a finish alone. A session event expires on its own.
+- **TE-146** A sweep counts the entries in a spool it could not lock, so a drain holding several
+  spools cannot keep the installation under the event cap.
+- **TE-147** A released version carrying build metadata (`1.2.3+linux`) is reported by name, which
+  the release workflow's own validator accepts.
+- **TE-148** A rollup that reads the generation, then the leases, then the generation again retries
+  when it moved, so a completed upload cannot be mistaken for a settled day.
+- **TE-149** Post-tool decision events are counted once: `executeAgentNode` is the single layer that
+  records them, and the capture equals the events actually emitted. A user-interrupted action's
+  test command is not recorded as task evidence.
+- **TE-150** The per-spool cap never evicts a run that is still open: an engine spool over the
+  event or byte limit sheds completed runs and session events oldest first, and a lone
+  `run.started` survives until its own `run.finished` joins it or the age bound removes the pair.
+  A spool made only of open runs may stay over the cap, because age is what discards them.
+- **TE-151** Concurrent decision requests are attributed per invocation: a retry's request does
+  not overwrite the questions of an earlier request still in flight, a completion for one
+  invocation leaves the other's count intact, and a later fallback for the other invocation
+  counts its own questions toward `fallback` and `unavailable`. The per-invocation entries survive
+  a capture merge.
+- **TE-152** A Paid opt-out is not acknowledged until its cached plan entry is durable: when the
+  telemetry state write fails, `config.set` still saves the configuration and emits the change but
+  answers `busy`, and a retry after the filesystem recovers stores `preference: "off"`. A failure
+  to cache an opt-in is not an error, because turning reporting on needs no tombstone.
+
+## Edge cases and potential bugs
+
+- **A breakdown must not contradict its total.** Bounds alone admit `toolCalls: 1` beside rows
+  summing to millions; the validator rejects the record rather than reconciling it.
+- **A retried upload must not create a second row.** The object key is the event's own id and date;
+  overwriting is prevented by the conditional create, which also protects the 13-month expiry.
+- **A partitioned rollup from an incomplete scan.** A multi-page listing can miss an object inserted
+  into a page already read; the before/after generation check is what prevents replacing a complete
+  rollup with an incomplete one.
+- **A writer that stops consuming.** SQS moves a message to the dead-letter queue only after a
+  bounded number of receives; a message never received expires at the source. The source queue's
+  depth and oldest-message age alarms exist for that case.
+- **Clock skew and wrong clocks.** A future timestamp is allowed five minutes; a past one thirty
+  days, which is also the outbox's discard bound.
+- **Concurrent engines.** The repository runs one engine per concurrent task against the same home.
+  Each owns its spool; a shared queue file without a protocol would lose entries.
+- **Big workspaces.** The workspace scan is bounded and skips dependency/build directories; it must
+  not add noticeable startup time to a task.
+
+## Gaps and limitations
+
+- **Nothing is deployed by this change.** The template and handlers are tested locally; the
+  `TelemetryApiUrl` output has not produced a live endpoint, and `src/auth/deployment.ts` ships an
+  empty `telemetryBaseUrl`. Until it is filled, the drainer is a no-op and events accumulate only
+  within the outbox bounds.
+- **The client's endpoint is wired to the deployed output by a documented step, not by this
+  merge.** `node scripts/telemetry-endpoint.mjs --write` reads the `TelemetryApiUrl` output from
+  `andorra-code-auth` and stamps it into `src/auth/deployment.ts`; the script requires the `/v1`
+  stage (giving a bare `execute-api` host that stage), refuses anything else and rerunning it is a
+  no-op. `ANDORRA_TELEMETRY_URL` overrides the stamped value for local runs and tests. A release
+  cannot ship an inert drainer: `release.yml` stamps the `TELEMETRY_API_URL` repository variable
+  when it is set and otherwise runs `--check`, which fails on an empty or invalid committed value.
+  See "Stamp the telemetry endpoint into the client" in `infra/andorra-auth/README.md`.
+- **No pricing table ships.** `costMicroUsd` and `pricingVersion` are absent at launch; every
+  figure is recomputable from token counts later.
+- **Browser-run trajectories do not emit `session.started` themselves.** The CLI and the app-server
+  do; a run driven only through the browser runner creates no conversation event of its own.
+- **Athena and dashboard queries were not exercised.** Rollups are written in the shape the ADR
+  describes; no dashboard reads them yet.
+
+## Validation evidence
+
+Automated checks run in this change (Linux/macOS, Node 22, Python 3.9):
+
+- `npm run build` — passed.
+- `npm run check` — passed.
+- `node scripts/run-tests.mjs dist/test/browser-runner.test.js` — the new setup-failure pair case
+  (TE-106) passes; three cases fail identically on unmodified `main`.
+- `node scripts/run-tests.mjs dist/test/telemetry-schema.test.js dist/test/telemetry-state.test.js dist/test/telemetry-emitter.test.js dist/test/telemetry-capture.test.js dist/test/telemetry-workspace.test.js` — 55 cases covering TE-01…TE-22, TE-30, TE-47, TE-59, TE-69, TE-83, TE-85, TE-87, TE-108, TE-109, TE-111, TE-115, TE-117 and TE-119, including a real local HTTP listener for the drain path. `dist/test/graph-health.test.js` covers the probe reference that TE-67 relies on; `dist/test/graph-runner.test.js` (52/52) covers the action-node instrumentation behind TE-78, and `dist/test/cli-graph-health.test.js` runs 14/15 with one case that also fails on `main`. `dist/test/analytics.test.js` — 15/15 (TE-29, TE-32); `dist/test/membership.test.js` — 23/23 (TE-29, TE-37); `dist/test/run-records.test.js` — 37/37 (TE-60, TE-114); `dist/test/decision-active.test.js` — TE-61 verified in the code-Graph application case, alongside five cases that fail identically on `main`.
+- `python3 -m unittest discover -s infra/andorra-auth -p 'test_*.py' -v` — 158 cases (TE-96 and
+  TE-116 among them): the existing
+  account suite plus the telemetry handler cases covering TE-23…TE-27, TE-38…TE-43, TE-54, TE-55 and
+  TE-57.
+- `npm run ci:local -- --only telemetry` — typecheck, build, the telemetry suite (including
+  `dist/test/telemetry-endpoint.test.js` for TE-132), and the Python discovery run: all four steps
+  passed.
+- `node scripts/run-tests.mjs` over the telemetry group's expanded test list (the ten per-file
+  suites plus the endpoint test) reports 16 failures in this environment: the six `cli.test.js` and
+  three `browser-runner.test.js` cases reproduce identically at the pre-change baseline, and the
+  seven `decision-active`/`decision-engine` cases fail because CodeGraph is unavailable here (the
+  prompt says "CodeGraph did not return usable context"). No failure names the behavior added by
+  this change.
+- `node --test dist/test/cli.test.js` (Node 22) — TE-128 and TE-140's new cases pass; six cases fail
+  identically on the `a3c249e2` baseline. `node --test dist/test/graph-runner.test.js
+  dist/test/analytics.test.js dist/test/telemetry-schema.test.js` — 94/94, including TE-129,
+  TE-130 and TE-131. `node --test dist/test/app-server.test.js` — 103/113, byte-identical failure
+  list to the same baseline (TE-118 and the earlier app-server rounds included). TE-133 is a
+  workflow step (`release.yml`) and is not executed locally; its two commands are the same ones
+  the local runs exercise.
+- `node --test dist/test/telemetry-capture.test.js dist/test/telemetry-state.test.js
+  dist/test/analytics.test.js` (Node 22) — 38/38, including TE-150 (an open run provocatively the
+  oldest group in an over-cap spool survives; a spool of only open runs stays over the cap),
+  TE-151 (two concurrent invocations, one completion, then the other's timeout counts one
+  fallback and one unavailable question) and TE-152 (a read-only telemetry directory makes
+  `config.set` answer `busy` after saving the configuration, and the retry stores
+  `preference: "off"`). `node --test dist/test/app-server.test.js` in the same run reports the same
+  eight baseline failures and no others.
+
+- `npm run ci:local -- --all --strict` — every group passed except the two failures that also
+  fail on untouched `main`: `dist/test/app-server-workspace.test.js` (2 cases) and
+  `desktop/src/engine/bridge.test.ts` (1 case). `rust-shell`'s `cargo check` ran (cargo is
+  installed here). The browser-runner cases fail the same way on `main` (3 cases) and were not
+  counted as regressions.
+
+Manual run verification (macOS arm64, Node 24, disposable home, ephemeral ports):
+
+- A fake OpenAI-compatible provider and a capture listener were started on OS-assigned ports and
+  `node dist/src/cli.js --prompt "Say hello"` ran against them with
+  `ANDORRA_TELEMETRY_URL=http://127.0.0.1:<capture>`. The capture received exactly three bodies:
+  `session.started`, `run.started` and `run.finished`, the last two sharing one `runId` and one
+  `installId`, with `appVersion: "0.1.0"`, `plan: "unknown"`, `outcome: "failed"`,
+  `failureClass: "limit"`, one `models[]` tuple (`requests: 4`, input/output tokens summed) and no
+  key outside the ADR inventory. The run itself ended `exhausted` because the fixture provider
+  returns plain text where the graph's stop hook expects a handoff; the classification of that
+  outcome is exactly what the case asserts.
+- The spool directory was gone after the drain: the events were uploaded, not merely written.
+- The same command with a seeded confirmed-Paid plan cache and `analytics: { enabled: false }`
+  captured **0** events.
+- This run found and fixed TE-28: the version lookup resolved to `dist/package.json` and reported
+  `unknown`. A regression test now asserts the manifest version.
+
+Manual checks still pending: TE-27 on a real deployment (the template is validated by the local
+CloudFormation logical-id tests, not by `validate-template`), and a review of the first day's raw
+objects and rollup against a capture server.