@dreamdata_io/verify-tracking 0.1.0

package/README.md

@@ -0,0 +1,756 @@
+# @dreamdata_io/verify-tracking
+
+Scrapes a live website with Playwright and asks an AI model whether the
+Dreamdata Analytics Script is correctly installed — script presence,
+`window.dreamdata`, cookies, Google Consent Mode v2, and dataLayer wiring.
+
+## Install
+
+Two channels, same package. Pick by what is already on the machine.
+
+### On a machine with nothing on it (recommended)
+
+```bash
+curl -fsSL https://unpkg.com/@dreamdata_io/verify-tracking/scripts/install.sh | sh
+```
+
+Paste, answer the sign-in, done. The installer fetches everything the run
+needs — a pinned Node 24 if the machine has none, the package, the Claude Code
+CLI, and Chromium if there is no Google Chrome to drive — then walks you
+through the one-time Claude sign-in on the same terminal. **No API key, no
+config file, no Node or npm required beforehand.**
+
+Everything lands in two places, and nothing else is touched:
+
+| Path | Holds |
+|---|---|
+| `~/.local/share/verify-tracking/` | the Node runtime, both packages, Chromium |
+| `~/.local/bin/verify-tracking` | the command (plus `verify-tracking-mcp`) |
+
+No `sudo`, no global npm prefix, and no system Node touched — an already
+installed Node 20+ or `claude` is reused rather than duplicated. Re-running the
+installer upgrades in place; `rm -rf` on those paths is a complete uninstall.
+Anything already provisioned is skipped, so a re-run after a failure is cheap.
+
+**The sign-in is the one step that cannot be unattended** — it is an OAuth flow
+through a browser, so a human has to finish it. The installer opens it for you
+rather than leaving you to discover the requirement on a failed first run. In a
+non-interactive shell (CI, a provisioning script) it installs everything and
+prints the one command left to run:
+
+```bash
+claude auth login
+```
+
+macOS and Linux, arm64 and x64. On Windows, install Node 20+ and use `npx`
+below.
+
+### On a developer machine
+
+Already have Node 20+ and a signed-in `claude`? Skip the installer:
+
+```bash
+npx -y @dreamdata_io/verify-tracking@latest --url https://example.com --quick --no-save-report
+```
+
+Three things are needed either way, and the installer is simply what supplies
+them. Each is checked at startup, so a missing one is reported in under a
+second rather than after the browser work:
+
+1. **Node 20+.**
+2. **The `claude` CLI, signed in.** The default backend shells out to a local
+   [Claude Code](https://claude.com/claude-code) CLI, so there is **no API key**
+   to distribute, rotate, or leak. Installed-but-logged-out is its own check
+   with its own fix.
+3. **A browser.** ~150MB of Chromium does not belong in an npm tarball, and
+   does not need to: the collector drives an installed Google Chrome first and
+   only falls back to Playwright's own Chromium, so a machine with Chrome on it
+   needs nothing extra.
+
+With `claude` signed in there is nothing to configure: no config file, no flags.
+
+### Using a hosted endpoint instead
+
+Put a key in `~/.config/verify-tracking/config.env` — a key is what selects
+that backend, so setting one is all it takes:
+
+```
+AI_API_KEY=...
+```
+
+That file exists because a globally installed CLI has no project `.env` to
+read. It is written once and is independent of the directory you run from. It
+only fills in what is unset, so flags and real environment variables still win
+over it. `VERIFY_TRACKING_OUTPUT_DIR` belongs here too if the default
+`verification/` tree should not land in whatever directory you happen to be in
+— see [Usage](#usage).
+
+A run is measured in minutes, not seconds: a single-phase `--quick` check took
+~2m45s, almost all of it waiting on the model. All three consent states take
+longer.
+
+## Setup for development
+
+```bash
+bun install
+bunx playwright install chromium
+cp .env.example .env   # fill in AI_API_KEY, or point AI_BASE_URL at a local model server
+```
+
+Bun is the development runtime; the published package is plain compiled JS and
+runs under Node (`engines.node >= 20`).
+
+### Cutting a release
+
+`.github/workflows/release.yml` publishes on a `v*.*.*` tag: it checks the tag
+against `package.json`, runs the typecheck and tests, and `prepublishOnly`
+rebuilds `dist/` so the tarball is never a stale checkout. Keep
+`src/version.ts` in step with `package.json` — the tag check does not see it.
+
+Publishing by hand does the same job:
+
+```bash
+npm publish --access public
+```
+
+Pick one per release. A manual publish followed by a tag push makes the
+workflow fail on a version already on the registry.
+
+## Usage
+
+One run, two commands:
+
+```bash
+bun run src/cli.ts --slug paytronix_com --url https://www.paytronix.com/
+bun run verification/review/build.ts
+```
+
+The first walks the site and **writes the report itself** — no output flag
+needed. It writes a pair, and prints both paths when the run ends:
+
+| File | What it holds |
+| --- | --- |
+| `verification/reports/<slug>_report.json` | The report. Every judgement, and the evidence behind it with the raw values that only fed those judgements taken back out. ~35KB — a size you can open, diff, or hand to a model. |
+| `verification/evidence/<slug>_evidence.json` | The same run, nothing trimmed: full request bodies, cookie values, the model's raw reply. Written for the archive, read when a judgement needs auditing. |
+
+So the command above writes `paytronix_com_report.json` and
+`paytronix_com_evidence.json`. Both come from one in-memory collection in one
+write step, so the pair is always the same run — re-running a live site never
+reproduces its evidence exactly, which is why the untrimmed copy is kept rather
+than re-derived on demand. `--out <path>` writes the report somewhere else and
+puts its companion beside it as `<path>.evidence.json`; `--no-save-report`
+skips both.
+
+Both paths are relative to the working directory, which is right inside this
+repo — the review dashboard reads exactly those two directories — and unwelcome
+when the CLI is installed globally and run from somewhere arbitrary. Set
+`VERIFY_TRACKING_OUTPUT_DIR` (a flag-free setting, so `~/.config/verify-tracking/config.env`
+is its natural home) to root them somewhere else; the run prints absolute paths
+when it ends either way.
+
+What the report leaves out, and why it is safe to leave out, is documented in
+`src/report/projection.ts`: every check and verdict is already resolved against
+the full evidence before serialization, so nothing the report drops could have
+changed one.
+
+The second regenerates the review dashboard's data from whatever is currently
+in `verification/reports/` — that directory decides which accounts appear, with
+`verification/evidence/` read on top of it for the raw values. It has to be
+re-run after every `cli.ts` run
+for `verification/review/index.html` to show the new report — there is no build
+step for the page itself; open the HTML file directly in a browser afterwards.
+
+That single `cli.ts` run already walks **all three consent states** — clean,
+grant, refused — each in its own fresh browser context, and reports them side by
+side. There are no consent flags to pass: `--phase clean,grant` narrows the set
+if you want fewer, and `--quick` is shorthand for `--phase clean`. Omit `--url`
+to be prompted interactively. See `bun run src/cli.ts --help` for all flags.
+
+`--slug` names the account the run belongs to. It sets the report filename and
+is recorded inside the file, so a run against a host that is not the account's
+own domain still files under the account:
+
+```bash
+bun run src/cli.ts --slug tungstenautomation_com --url https://sign-in.printix.net/
+bun run verification/review/build.ts
+```
+
+Without it the slug is derived from the URL's hostname — which would have filed
+that run as `printix_net` and lost the account. The review index flags any report
+whose filename and recorded slug disagree.
+
+See [Verification review index](#verification-review-index) below for what the
+dashboard shows and how it's structured.
+
+### AI backends
+
+- `--backend local-cli` — shells out to an installed AI CLI. `--cli-preset claude` is the default and the
+  only verified one; for anything else pass `--cli-cmd` with a command template using `{model}`/`{prompt}`
+  placeholders.
+- `--backend openai` — any OpenAI-compatible chat completions endpoint. Works unchanged for OpenRouter
+  (`--base-url https://openrouter.ai/api/v1`), a local model server (e.g. a local vLLM/Ollama/LM Studio
+  endpoint), or any other compatible provider — just change `--base-url`/`--model`.
+
+**An API key picks the backend.** With one set — flag, environment, or config
+file — the hosted endpoint is used; with none, the local CLI is, since it is
+the only one that can answer without credentials. `--backend` overrides the
+choice either way. An unconfigured run used to select the endpoint and die on a
+401 naming nothing the operator could fix, which is the wrong default for a
+tool whose intended setup carries no key at all.
+
+The default model follows the backend, because the two name models
+differently: `anthropic/claude-sonnet-4.5` for the endpoint, `sonnet` for the
+claude CLI. Do not cross them — `claude --model anthropic/claude-sonnet-4.5`
+is refused **and still exits 0**, so the refusal used to be read back as the
+model's answer. `local-cli` now rejects a zero-exit reply that carries no JSON
+rather than reporting a verdict it could not parse.
+
+### Running against a local model
+
+The `openai` backend is the local path too — a local server is just another
+base URL:
+
+```bash
+bun run src/cli.ts --slug clio_com --url https://www.clio.com/ \
+  --base-url http://localhost:11434/v1 --api-key ollama --model dd-verify
+```
+
+**Build the model tag first; do not point `--model` at a stock Ollama tag.**
+Ollama's default context is 4096 tokens; a three-phase run's prompt measures
+around 16k — a ~6k system prompt (the checklist and shape guidance) plus the
+evidence view — so a stock tag silently truncates the payload and then scores
+checklist items against evidence it was never shown, a wrong verdict that
+reports as a clean run. `replay.ts` prints the measured count for the run in
+front of you. The `Modelfile` at the repo root pins a context window that fits:
+
+```bash
+ollama create dd-verify -f Modelfile
+```
+
+LM Studio works the same way at `http://localhost:1234/v1`.
+
+**Check the model actually fits.** `ollama ps` reports a `PROCESSOR` column: a
+model larger than free RAM runs partly on the CPU, and the difference is not
+subtle. Measured on a 24GB machine, the same single-phase prompt took 148s on
+`qwen2.5-coder:14b` (fully on GPU) and had not finished after ten minutes on
+`qwen2.5-coder:32b` (28GB required, 38% on CPU). Waiting for the latter is a
+choice you can make — `--ai-timeout` exists for it, and defaults to 30 minutes
+— but a batch of accounts is where that cost lands.
+
+### Replaying a saved run against another backend
+
+Comparing two backends by re-running `cli.ts` against a live site compares two
+different observations: the site changed, the banner behaved differently, a bot
+wall appeared. `verification/replay.ts` removes that variable — it rebuilds
+`MultiPhaseEvidence` from a saved report, re-runs the deterministic
+`analyzeDeployment` over it, and sends that same fixed evidence to whichever
+backend you name:
+
+```bash
+bun run verification/replay.ts --slug clio_com \
+  --base-url http://localhost:11434/v1 --model dd-verify --compare
+```
+
+`--compare` diffs the result against the verdicts already in that report and
+exits non-zero on a regression. What it checks, in the order that matters:
+
+1. **`overallPass` did not flip** — that is the exit code, and it must not
+   depend on which model was asked.
+2. **Every checklist id is present exactly once**, with no invented ids.
+3. **Each `passed` matches.** A `true` ↔ `false` flip fails the run;
+   `inconclusive` ↔ `false` is tabled for a human but tolerated.
+4. **`evidenceQuote` names a field that actually exists** in the evidence or the
+   analysis. Inventing a field path is the failure mode small models reach for,
+   and `checker.ts`'s prompt forbids it explicitly.
+5. **`remediationAdvice` does not tell an installed customer to install the
+   script** when `deployment.shape` already confirms an install.
+
+The replayed run is written to `verification/reports/local_ai/` — beside the
+baseline it read, never over it (`--out-dir` moves it, `--no-save-report` skips
+it). To look at those runs in the dashboard:
+
+```bash
+bun run verification/review/build.ts --reports-dir verification/reports/local_ai --out verification/review/local-data.js
+```
+
+then open `verification/review/index.html?data=local-data.js` — the same page,
+which reads its dataset from that query parameter and labels its own header
+with the reports directory it loaded, so a replayed run is never mistaken for a
+ground-truth one.
+
+## The three consent states
+
+Every consent verdict is a **comparison**, not a property of one page load: a
+bundle that is correctly gated and one that is broken both look like silence. So
+each run observes three states, each in its own fresh browser context (the
+isolation matters — a grant leaves a cookie that would make the next state replay
+the granted page):
+
+| Phase | What it does |
+| --- | --- |
+| `clean` | Loads the page and leaves the banner alone |
+| `grant` | Drives the banner's accept control, waits for it to persist, reloads |
+| `reject` | Drives the banner's reject control, waits for it to persist, reloads |
+
+`--phase clean,grant` runs a subset; `--quick` is shorthand for `--phase clean`.
+
+## The second page
+
+Each phase then follows one internal link, because a first page view is the one
+page view a broken install still tends to get right. Two failures live entirely
+in the gap between page one and page two:
+
+- A consent-gated bundle injected **once**, on the load that followed the grant,
+  and never again. The visitor accepted, got identified tracking for exactly one
+  page, and was quietly handed back to the cookieless fallback after that — so
+  their session arrives split in two, half identified and half anonymous.
+- A **single-page app** whose router swaps the route without a document load. No
+  script re-runs, so a bundle that only tracks on load never sends another page
+  event no matter how many pages the visitor reads.
+
+Both look perfect in every first-page field, which is why the second page's
+evidence is kept separate (`phases[].secondPage`) rather than folded into the
+phase's own counts.
+
+The link is picked in the page and clicked the way a visitor would click it —
+never with `page.goto`, which forces a document load and would hide the SPA case
+outright. `secondPage.kind` records what the site did with it: `hard` (a new
+document) or `soft` (the router handled it in the same document, with the
+`pushState` to prove it). Legal, auth and asset URLs are skipped
+(`SECOND_PAGE_PATH_DENYLIST`) since a missing request there says nothing about the
+install, and the click is dispatched on the element so an unanswered consent
+banner cannot swallow it.
+
+When the walk is handled by the router rather than by a document load, the
+phase is left with **no unambiguous page view at all**: its entry page was
+loaded twice around the consent answer, so two page events there can be the
+consent upgrade rather than two loaders, and the page it routed to never
+re-ran a script. So a soft walk is followed by one plain document load of the
+page it landed on (`secondPage.hardLoad`) — the same navigation a visitor
+makes when they open that page directly or refresh it.
+
+That load is what `page-view-counted-once` can actually judge. meteomatics.com
+is the case it was added for: every internal link routes in-document, and a
+plain load of the routed-to page sends **both** bundles' page events — 6 of 8
+warm-cache loads, 20–70ms apart, on a settled consent decision with no answer
+anywhere near it. Nothing else the run observed could say so.
+
+It costs one reload and one flush wait per phase, and only in phases whose
+walk was soft; a walk that already landed a document load records
+`hardLoad: null`.
+
+`--no-second-page` skips the walk. Then the three checks below read
+`inconclusive`, and the run sees one page view per phase again.
+
+The comparisons are then made in code, not by the model:
+
+| Check | Expectation |
+| --- | --- |
+| `clean-before-consent` | No normal request and no identity cookie while the banner is unanswered |
+| `grant-enables-normal` | Accepting must actually start identified tracking |
+| `reject-blocks-normal` | Refusing must actually stop it |
+| `cookieless-covers-unconsented` | Where nothing else tracks, the cookieless bundle does |
+| `grant-sets-identity-cookies` | A running normal bundle stores an anonymous id |
+| `reject-writes-no-identity` | A refusal leaves no cookie or localStorage entry behind |
+| `consent-mode-signal` | `analytics_storage` follows the choice, with source `update` |
+| `second-page-keeps-tracking` | After a document load onto another page, whatever was tracking still is |
+| `soft-nav-keeps-tracking` | After an in-app route change, a page event still fires |
+| `second-page-honours-consent` | A navigation does not turn identified tracking on by itself |
+| `page-view-counted-once` | One page view produces one `page` event, not two (the banner-answering entry page is exempt: it loaded twice) |
+
+The two continuity checks split one question by where its answer lives: a
+document load that stops tracking is a missing bundle or tag-manager trigger on
+non-entry pages, while a route change that stops tracking is a bundle that is
+right there and a router that never calls `page()`. Every phase falls into
+exactly one of them, so nothing is counted twice.
+
+A failed check sinks the run regardless of what the model says. A check whose
+phase was not run is `inconclusive`, never a failure.
+
+## Deployment shapes
+
+Dreamdata installs legitimately differ, and all of these are healthy. The shape is
+computed from all phases at once — in code, not inferred by the model — and handed
+to the checker as a fact, so a valid shape is not reported as a defect:
+
+| Shape | Looks like | Cookies |
+| --- | --- | --- |
+| `cookieless-by-design` | Only a cookieless bundle exists anywhere | none, by design |
+| `consent-gated-dual` | Both bundles belong to the install; consent picks | `dd_*` once granted |
+| `normal-only` | Only the standard bundle is installed | `dd_*` |
+| `helper-only` / `nothing-installed` | No tracking bundle at all | — a real failure |
+| `undetermined` | Would read as `nothing-installed`/`cookieless-by-design`, but a detected consent banner could not be driven | can't tell — needs a manual click-through |
+| `third-party-segment-undetermined` | Would read as `nothing-installed`/`helper-only`, but a REAL Segment install (`cdn.segment.com` script and/or a request to `api.segment.io`) was found instead of a Dreamdata bundle | can't tell — Segment→Dreamdata forwarding is server-side and invisible to this harness; check Segment's destination config or Dreamdata's ingestion records directly |
+
+`deployment.segmentCompat` is orthogonal: the install exposes `window.analytics`
+and `ajs_*` storage instead of `window.dreamdata` and `dd_*`. Also valid.
+
+### Why "no normal bundle in the DOM" proves nothing
+
+`cookieless-by-design` is the shape that most often gets asserted wrongly, because
+a consent-gated or tag-manager-injected bundle is **absent from the DOM by
+design** — exactly when the question matters. So the shape is only claimed when
+none of these signals is found in **any** phase (`script.normalBundleSignals`
+records whichever did):
+
+| Signal | What it means |
+| --- | --- |
+| `request` | A normal-mode request was actually sent |
+| `dom-script` | A normal bundle tag is in the DOM |
+| `cmp-blocked-script` | The tag is there but neutralised (`type="text/plain"`, `optanon-category-*`, …) — installed, not absent |
+| `tag-manager-container` | The page's published GTM container ships the tag, with the consent types it is gated on |
+| `cookieless-loader-companion` | The cookieless loader checks for `#dreamdata-analytics`, i.e. it expects a normal bundle |
+| `stub-queue` | `window.dreamdata` is the loader's stub array — calls queued for a bundle that never came |
+
+The container scan is one HTTP fetch per container per run, cached across phases;
+`--no-tag-manager-scan` skips it, at the cost of reading a GTM-only install as
+absent.
+
+Three consequences worth knowing:
+
+- **Cookieless requests carry no `anonymousId`.** That is the point of the mode —
+  identity resolution happens server-side. Only `writeKey` is required.
+- **`window.dreamdata` can be a stub array.** A loader snippet creates its global
+  as an array that queues calls until the bundle replaces it, so an array means
+  the bundle never arrived (`globals.dreamdataIsStubQueue`). Expected when it is
+  consent-gated; a real defect when it is not.
+- **The two bundles are mutually exclusive, not ordered.** The cookieless snippet
+  stands down when it finds `#dreamdata-analytics` present and still
+  `type="text/javascript"` (a CMP that blocks the tag rewrites that type, which
+  is what lets cookieless take over), and the normal bundle `delete`s
+  `window['dreamdata-cl']` as soon as it initializes. So a missing cookieless
+  global in a granted phase is the handoff working, and both modes tracking *one
+  page view* is the finding — that page view counted twice, usually because the
+  cookieless snippet sits above the normal one. Scoped to a page view, not to a
+  phase: `page-view-counted-once` counts `page` requests per observed page view,
+  and it skips the entry page of a phase that answered the banner, because the
+  run loaded that URL twice — anonymous before the answer, identified after —
+  which is the consent upgrade rather than a duplicate loader. Which mode fired
+  first is a race with no correct answer, so `network.cookielessRequestSeenFirst`
+  is reported for humans but never scored.
+
+## Headless runs advertise themselves, and sites answer differently
+
+Headless Chrome writes `HeadlessChrome/<version>` into its User-Agent — in the
+request header and in `navigator.userAgent` alike — and `isbot`-style checks
+match that substring. Sites do not merely *log* such a visitor as a bot; they
+change what they render. meteomatics.com never mounts its consent banner at
+all for a matching UA, so every phase found no CMP, no accept control and
+nothing to drive, and all three consent states measured the same banner-free
+page — a page no visitor is ever served.
+
+So a headless run renames it back to `Chrome`, which is what the same binary
+reports when it has a window, and does it at the browser-context level so the
+request header changes too and not just the JS-visible copy a server-side
+check never reads. Headed runs (`--headed`) are left alone.
+
+Worth knowing when a run looks implausible: this is one specific tell, not a
+stealth suite (see `stealth.ts` for the rest), and a site behind stronger bot
+management can still serve automation a different page.
+
+## Consent-gated sites
+
+A CMP running in automatic blocking mode (e.g. Cookiebot with
+`data-blockingmode="auto"`) rewrites tracking tags to `type="text/plain"` before
+the visitor answers the banner, so nothing loads and nothing is tracked. **That is
+correct GDPR behaviour, not a broken install** — and the `clean` phase reports it
+as such, while `grant` shows what the same page does once consent exists.
+
+Consent state is reported with its provenance. `analytics_storage: denied` from
+Google's own implicit default is *not* a user decision; check
+`consent.analyticsStorageSource` before reading a denial as one.
+
+Two failure modes worth naming, both of which need more than one state to see:
+
+- **The CMP stores a choice Consent Mode never receives.** The banner records the
+  answer, `analytics_storage` never moves, and every gated tag stays frozen.
+  `consentAcceptance.consentSignalConfirmed` is false and `consent-mode-signal`
+  fails — distinct from a broken bundle, and fixed somewhere else entirely.
+- **A tag manager holds the bundle back for its own reasons.** Consent is granted,
+  the Consent Mode signal lands, and the tag still never fires because its
+  trigger does not cover the page. `grant-enables-normal` fails and says so.
+
+## First-party tracking hosts
+
+Dreamdata-owned hosts are detected automatically, matched on the request's
+**hostname** so a third party's `?referer=dreamdata.io` query string cannot pass
+as an install. A first-party install on `api.s.dreamdata.io` therefore needs no
+configuration. Only a fully custom CNAME (`track.yourdomain.com`) has to be
+named with `--tracking-host`.
+
+Worth knowing that a first-party install spans **three distinct hosts**, and the
+one that matters for requests is the delivery host — not the CDN in `_cdn`:
+
+| Role | Example on dreamdata.io |
+| --- | --- |
+| Loader script | `evs.s.dreamdata.io/…/CxqWbtgYml.min.js` |
+| Bundle CDN (`analytics._cdn`) | `evs.s.dreamdata.io` |
+| **Event delivery** | **`api.s.dreamdata.io/v1/p`** |
+
+```bash
+bun run src/cli.ts --url https://dreamdata.io \
+  --tracking-host api.s.dreamdata.io
+```
+
+`--tracking-host` repeats and accepts comma-separated values; `DD_TRACKING_HOSTS`
+sets it via env. The watched set is recorded in `network.trackingHosts`, so a run
+that captured nothing shows what it was looking at.
+
+Delivery paths are `/api/v1/<letter>` on the shared CDN and `/v1/<letter>` on
+first-party hosts; both are recognised.
+
+## Manual verification
+
+No fake fixture site is used. Sanity-check against a site you can inspect in
+DevTools, and compare field by field:
+
+| Evidence field | DevTools check |
+| --- | --- |
+| `script.present` / `writeKey` | `document.querySelectorAll('script[src*="dreamdata"], script[src*="drda"]')` |
+| `network.requests` | Network tab filtered on the host in `network.trackingHosts` |
+| `globals.*` | `window.dreamdata`, `window['dreamdata-cl']`, `window.analytics` |
+| `cookies.cookieNames` | Application → Cookies, filter `dd_` (or `ajs_`) — the report lists which are set, `verification/evidence/` has their values |
+| `consent.*` | `window.google_tag_data.ics.entries.analytics_storage` |
+| `secondPage.*` | Click an internal link yourself and watch the Network tab; `secondPage.kind` should match whether the tab reloaded |
+
+To diff raw values field by field, read the run's
+`verification/evidence/<slug>_evidence.json` — that is the untrimmed copy, and
+it is already on disk from the run you just did.
+
+## What the verdicts are, and are not
+
+Evidence collection, the deployment shape and the cross-phase consent checks are
+deterministic code. The **checklist verdicts are an LLM reading that evidence**,
+so their wording varies between runs on the same page; the model is told the shape
+and the resolved checks as facts rather than being asked to derive them, because a
+single misread boolean used to turn every consent check into "not applicable" and
+the report into a green PASS.
+
+`overallPass` is not taken from the model either. It is derived from the per-item
+verdicts — any `false` sinks the run, `not_applicable` and `inconclusive` do not,
+and an all-inconclusive run is not a pass — and then a failed cross-phase check
+sinks it regardless. The process exit code follows it.
+
+Each cross-phase check also carries `phases`: the consent states its status
+actually rests on, and for a **failure** only the states that actually failed. A
+continuity check that spans three phases and broke in one says so in one field
+rather than only in its prose, so anything drawing a per-state view — the review
+index's matrix — can attribute the failure instead of painting all three red.
+
+## Driving the CLI from an MCP client
+
+The CLI also runs as an [MCP](https://modelcontextprotocol.io) server, so an
+agent can run a verification and then read the result:
+
+```bash
+bun run src/mcp/server.ts
+```
+
+| Tool | What it does |
+| --- | --- |
+| `verify_tracking` | Runs the full three-state check and writes `verification/reports/<slug>_report.json`. Returns a summary and that path — never the report itself, which runs 40–50KB. |
+| `list_reports` | Every saved report, newest first, with its shape and verdict. |
+| `read_report` | One report by slug, or one field of it (`deployment.checks`, `phases[0].script`). |
+| `explain_verdict` | One checklist item, with the field its `evidenceQuote` names resolved — so a quote pointing at a field that does not exist shows up as `resolved: false` — plus the cross-phase consent checks. |
+
+The backend is read from `.env` (or the environment), never from a tool
+argument: a tool call is model-authored text, and a call that could name its
+own `--base-url` could send a run's whole evidence payload to a host of the
+model's choosing.
+
+### Pairing it with chrome-devtools-mcp
+
+The reason to run this over MCP is having both servers in one agent —
+[`chrome-devtools-mcp`](https://github.com/ChromeDevTools/chrome-devtools-mcp)
+for looking, this one for measuring:
+
+```jsonc
+{
+  "mcpServers": {
+    "verify-tracking": {
+      "command": "bun",
+      "args": ["run", "src/mcp/server.ts"],
+      "cwd": "/path/to/tracking-verification-cli",
+      // A full run is minutes, not seconds — see "One run is a long tool call" below.
+      "timeout": 900000
+    },
+    "chrome-devtools": { "command": "npx", "args": ["chrome-devtools-mcp@latest"] }
+  }
+}
+```
+
+Goose, Cline and mcphost all take this shape; only the file it goes in
+differs. Ollama and LM Studio are model runtimes, not MCP clients — a local
+model reaches these tools through a client like those, not on its own.
+
+For Goose specifically, `verification/goose-config.example.yaml` is that block
+already filled in, and `.goosehints` in the repo root is what the agent is told
+about the two servers. That file is not optional decoration: without it a model
+will re-check a consent verdict by driving one browser tab through accept and
+then reject, which measures a page still carrying the previous answer's cookie.
+Whatever client you use, give it those instructions in whatever form it reads.
+
+Two things to know before pointing a local model at an agent loop:
+
+- **Tool calling is required.** `ollama show <model>` must list `tools` under
+  Capabilities. `qwen2.5-coder:14b` does.
+- **An agent loop spends context that single-shot scoring does not.** The
+  Modelfile pins `num_ctx 65536`, but qwen2.5-coder's trained context is 32768
+  — fine for one ~16k-token scoring prompt, less so for a conversation
+  accumulating tool results. This is why the tools return summaries and a field
+  path rather than reports: `read_report` without a `path` will spend 40-50KB of
+  that budget in one call.
+
+A run then looks like: `verify_tracking` on the URL → read the failing item
+with `explain_verdict` → `navigate_page` and `list_network_requests` in
+DevTools to look at the actual request that item is about.
+
+### One run is a long tool call
+
+`verify_tracking` takes minutes: three consent states at roughly 15–20s of
+page loading each, and then a model reading a ~16k-token prompt — which on a
+local 14B model measured 148s for a single-phase run, and grows with the
+phases. Most clients time a tool call out well before that.
+
+The server sends a progress notification every 10 seconds while a run is going,
+naming the step it is on. That is enough for a client that resets its timeout
+on progress — in the MCP TypeScript SDK that is `resetTimeoutOnProgress`, which
+is **off** by default, so a client has to opt in.
+
+**Claude Code's timeout is not one of them**: its per-server `timeout` (and the
+`MCP_TOOL_TIMEOUT` environment variable it overrides) is a hard wall-clock
+limit per tool call that progress notifications do not extend. Set it to cover
+a whole run — the `"timeout": 900000` above is 15 minutes — rather than relying
+on the heartbeat.
+
+**Measure with this server; look with DevTools — not the other way round.** An
+agent that "re-checks" a consent verdict by driving one browser tab through
+accept and then reject is measuring a page that still carries the previous
+answer's cookie. Every consent verdict here is a comparison between three
+states collected in three *fresh* contexts (see
+[The three consent states](#the-three-consent-states)); a shared tab cannot
+reproduce it, and what it produces instead is a confident contradiction of a
+correct report.
+
+## Verification review index
+
+`verification/review/index.html` gives a browsable view of every report in
+`verification/reports/`. One `<slug>_report.json` there is one account on the
+page; nothing else is aggregated, so whatever `cli.ts` has written is exactly
+what the dashboard shows.
+
+It is built in three levels, so the first screen is scannable and everything
+that needs reading is one click away:
+
+| Level | Shows |
+| --- | --- |
+| **The list** | One line per account: slug, overall PASS/FAIL, the check tally (`✕` fail / `?` inconclusive / `✓` pass), and a 3×5 thumbnail of that account's matrix. |
+| **The card** | The **matrix** — three consent states down the side, five signals (`script present`, `event on page load`, `event on page nav`, `identity storage`, `consent mode`) across the top, one verdict per cell — then every cross-phase check as a single line, worst first. |
+| **The reason** | A cell opens the raw values it was computed from plus the check that judged it; a check line opens its own detail and evidence. *Everything measured* holds the complete per-phase evidence JSON, the deployment analysis and the model's raw response — `build.ts` reads those from `verification/evidence/` when the run archived them, so nothing is summarised away here even though the report file itself is trimmed. |
+
+Cell colour is never re-derived in the page. Every consent comparison is already
+resolved in `src/analysis/deployment.ts` and shipped as `deployment.checks`; a
+cell takes the worst status of the checks that name its phase, and `script
+present` is left uncoloured because a tag's presence is a fact about the install
+(a consent-gated tag is absent by design exactly when the question matters), not
+a verdict. A **dashed** cell is one whose colour is genuinely shared by more than
+one consent state.
+
+Rebuild its data after any sweep run:
+
+```bash
+bun run verification/review/build.ts
+```
+
+This writes `verification/review/data.js`, which `index.html` loads directly
+via `<script src="data.js">` (avoiding `fetch()`/CORS restrictions when opened
+via `file://`). Re-run it any time a report under `verification/reports/`
+changes, then open `index.html` in a browser.
+
+## Known limitations
+
+- Each phase covers two page views — the load, then one followed link — plus a
+  document load of the second page when the walk was soft. Form submissions are
+  still never driven, so the `event-behavior` item stays inconclusive on that
+  sub-point. Only one link is followed, and only from the entry page, so a site
+  that breaks on the third page or on one particular section will not be caught.
+- A page view whose two loaders race is only sometimes doubled, so one
+  observation of a single page event does not clear a site. On meteomatics.com
+  the same URL sent one request on 5 of 6 cold-cache loads and two on 6 of 8
+  warm-cache ones — a tag manager injecting the normal bundle either side of
+  the moment the cookieless snippet looks for it. A run sees one load per
+  phase and can therefore miss it; a pass on this check is weaker evidence
+  than a failure.
+- The entry page of a phase that answered the banner is never judged on
+  double-counting unless its page events name two different builds. Two loads
+  of one URL either side of a consent answer are two runs of one install, and
+  the ordering of the two bundles cannot separate that from a real duplicate:
+  on paytronix.com the normal event the accept click produced is stamped
+  mid-reload and so arrives *before* the reloaded page's cookieless one,
+  which looks exactly like the race but is not. The extra document load above
+  exists because inference on that page view does not work.
+- The second-page probe picks the link itself. A site whose entry page has no
+  followable same-origin link, or whose link opens a modal instead of
+  navigating, records `urlChanged: false` and leaves the three continuity checks
+  inconclusive rather than failed.
+- A known CMP is driven via its vendor API or a vendor-specific selector; add the
+  vendor to `CMP_VENDORS`/`DOM_ONLY_CMP_VENDORS` and the selector lists in
+  `consent-accept.ts` to cover a new one. An unrecognised CMP falls back to a
+  heuristic: a clickable control whose exact text matches a curated accept/reject
+  phrase, inside a container that both mentions cookies/consent/privacy and holds
+  the opposite-direction control too (`heuristic-banner.ts`) — this is what stops
+  an unrelated page button (e.g. a signup form's "Accept") from being mistaken for
+  a consent banner.
+- A banner with no reject control anywhere is still *detected*: the heuristic
+  proves a container is a consent banner by finding a second consent control
+  next to the one it matched, and where there is no opposite-direction control
+  a cookie-explicit settings link plays that part
+  (`PREFERENCE_CENTER_COUNTERPART_PHRASES` — the vaguer "settings"/
+  "preferences" openers are excluded, since they sit in too many page footers
+  to prove anything). Requiring the opposite control outright made those
+  banners invisible, which in turn meant the preference-centre flow written to
+  answer them was never reached.
+- Banners with no one-click answer in the direction we want — the common
+  "Accept all" + "Cookie settings" pair, with no reject button anywhere — are
+  answered through their preference centre instead (`preference-center.ts`):
+  open the panel, set every non-essential category toggle to the target state,
+  then click the panel's own save control. The save search deliberately excludes
+  accept-all/reject-all shortcuts, which would discard the toggles, and both the
+  open and save searches are scoped the same way the accept/reject heuristic is —
+  the settings link must sit in a cookie-context container that also holds a real
+  accept/reject control, and the save control must sit around the category
+  toggles themselves. Always-on essential categories (matched by
+  `ESSENTIAL_CATEGORY_PATTERN`, or simply disabled) are left alone. Add a vendor
+  to `PREFERENCE_CENTER_OPEN_SELECTORS`/`PREFERENCE_CENTER_SAVE_SELECTORS` to
+  drive its panel by element rather than by button text. The category-toggle
+  search is scoped more tightly than the banner search is: a container only
+  counts as the panel when it mentions cookies or consent — not merely
+  "privacy", which every marketing form's terms checkbox does — and is small
+  enough to be a panel rather than the page (`PREFERENCE_PANEL_*`). Without
+  both, an unrelated form's checkboxes join the toggle set, the panel resolves
+  to the whole document, and the save-button search is scoped to it.
+- When none of the above finds or clicks a control, the
+  `grant`/`reject` drive is recorded as not-succeeded; if a CMP was detected but
+  every attempted drive failed, a would-be `nothing-installed`/`cookieless-by-design`
+  shape is downgraded to `undetermined` instead of asserted, and the affected
+  checks read `inconclusive` rather than failed — honest, but not a verdict.
+- The GTM consent gate is read out of a minified container by looking backwards
+  from the bundle filename for the nearest `"consent":["list",…]`. It is reported
+  as a corroborating signal, never as the sole basis for a verdict.
+- Debug mode (`dreamdata.debug(true)`) is never actually enabled.
+- `console.errors`/`warnings` are filtered to messages mentioning "dreamdata", so
+  an empty list means no Dreamdata-specific error, not an error-free page.
+- Anonymous-id stability is only partly observable: the id on the first page and
+  the one in the same phase's `secondPage` can be compared, but no check asserts
+  on it — the model reports it, and a drift there is worth a manual look. The
+  report itself lists only which entries were set, so read the values out of
+  `verification/evidence/<slug>_evidence.json`.
+- A site that reassigns `window.dataLayer` after the hook installs is covered —
+  the accessor re-wraps whatever is assigned — but a snippet holding a direct
+  reference to the array from before the hook can push past the wrapper.
+- A bespoke banner's decision is only seen to persist when it lands in a cookie
+  or localStorage entry that is either new or *named* like a consent record (a
+  vendor prefix, or a name containing "consent"/"cookie settings" and friends).
+  A banner that records its answer under an unrecognisable name it also wrote
+  on load reads as `not-persisted`, and the phase is reported as not driven
+  even though the click worked.