token-goat 2.6.36 → 2.8.1

package/README.md

@@ -287,7 +287,7 @@ For recurring scheduler loops, the 25th, 100th, and 250th observed delivery in a
 
 > **Easiest install:** paste this repo's URL into your AI and ask it to install token-goat properly. It will run the commands, check codecs, and confirm everything is working.
 
-**Requirements:** Node.js 22 or later (all platforms)
+**Requirements:** Node.js 22.16 or later (all platforms)
 
 ```
 npm install -g token-goat
@@ -305,7 +305,7 @@ Installation adds a short routing guide to the agent's instructions. When the ag
 
 | Task | Agent flow |
 | --- | --- |
-| Review a PDF | `pdf-meta` and `pdf-outline`, then `pdf-extract --pages` |
+| Review a PDF | `pdf-meta` and `pdf-outline`, then `pdf-locate` to find the pages that mention a term and `pdf-extract --pages` only those |
 | Review a Word document | `docx-outline`, then `docx-text` |
 | Review a slide deck | `pptx-outline`, then `pptx-slide` or `pptx-notes` |
 | Review a workbook | `xlsx-sheets`, then `xlsx-head`, `xlsx-range`, or `xlsx-query` |
@@ -451,7 +451,9 @@ token-goat install --copilot
 
 The `--copilot` flag patches Claude Code and registers a Copilot CLI hook config: `~/.copilot/hooks/token-goat.json` (a `{ version, hooks }` file registering `sessionStart`, `preToolUse`, `postToolUse`, `preCompact`, `agentStop`, `subagentStop`, and `userPromptSubmitted`, per Copilot's own [hooks reference](https://docs.github.com/en/copilot/reference/hooks-reference)) plus the shim script it points at, `~/.copilot/hooks/token-goat-shim.js`. Unlike Codex, Copilot's event names and response schema (`permissionDecision`/`modifiedArgs` for `preToolUse`, `modifiedResult`/`additionalContext` for `postToolUse`, `decision`/`reason` for `agentStop`/`subagentStop`) genuinely differ from Claude Code's, so the shim translates rather than passes through.
 
-What works: **the command-routing reminder** (`sessionStart` returns `additionalContext`, so Copilot is told token-goat exists before it picks its first read tool — this is the one channel that lands ahead of that decision), **bash output compression and re-read denial** (`preToolUse` returns `modifiedArgs` or `permissionDecision: "deny"`), **image shrinking and post-edit indexing** (`postToolUse` returns `additionalContext`), and **stop-hallucination logging** (`agentStop`/`subagentStop` map a token-goat `deny` onto `decision: "block"`, everything else onto `decision: "allow"`). `preCompact` and `userPromptSubmitted` are notification-only on real Copilot CLI, per its docs: Copilot never reads a response body for either, so token-goat's compaction manifest and prompt-context hints have no surfacing channel there. The shim still calls through for both so token-goat's internal side effects keep running, but nothing gets injected back into the agent. Copilot's built-in tool names (`view`, `edit`, `create`, `bash`/`powershell`, `web_fetch`, `grep`, `glob`, `memory`, and MCP-server calls) are remapped onto token-goat's internal names where a clear match exists (`view`→Read, `edit`→Edit, `create`→Write, `bash`/`powershell`→Bash, `web_fetch`→WebFetch, `grep`→Grep, `glob`→Glob); `memory`, `task`, `ask_user`, and MCP tool calls pass through unmapped and simply no-op.
+What works: **the command-routing reminder** (`sessionStart` returns `additionalContext`, so Copilot is told token-goat exists before it picks its first read tool — this is the one channel that lands ahead of that decision), **bash output compression and re-read denial** (`preToolUse` returns `modifiedArgs` or `permissionDecision: "deny"`), **background-shell output compression** (`postToolUse` returns `modifiedResult`), **image shrinking and post-edit indexing** (`postToolUse` returns `additionalContext`), and **stop-hallucination logging** (`agentStop`/`subagentStop` map a token-goat `deny` onto `decision: "block"`, everything else onto `decision: "allow"`). `preCompact` and `userPromptSubmitted` are notification-only on real Copilot CLI, per its docs: Copilot never reads a response body for either, so token-goat's compaction manifest and prompt-context hints have no surfacing channel there. The shim still calls through for both so token-goat's internal side effects keep running, but nothing gets injected back into the agent. Copilot's built-in tool names are remapped onto token-goat's internal names where a clear match exists (`view`→Read, `edit`→Edit, `create`→Write, `bash`/`powershell`→Bash, `read_bash`/`read_powershell`→BashOutput, `web_fetch`→WebFetch, `grep`→Grep, `glob`→Glob). MCP-server tool calls, which Copilot names `<server>-<tool>` rather than `mcp__<server>__<tool>`, are translated too, but only when the name matches Copilot's own cached tool list exactly — never guessed from the name's shape, because a server name can itself contain a hyphen and a wrong guess would make the read-only MCP dedup path deny an ordinary built-in call. With no cache to match against, nothing is translated. `memory`, `task`, `ask_user`, `write_bash`/`write_powershell` (which send keystrokes to a running shell, not commands), and `stop_bash`/`list_bash` pass through unmapped and simply no-op.
+
+**Why the background-shell compression matters most on Copilot.** Copilot runs shell commands in the background: a build or a test suite is started once, and the model then checks on it repeatedly while it runs. Each check hands back everything the command has printed since it started, from the first line. So the second check re-sends the whole first check, the third re-sends the first two, and a check ten minutes into a slow build re-sends the same output for the tenth time. The model has already read all of it and pays again for every word, every time. Token-goat sends the first check through untouched, then returns only the new part on each later check, with one line saying that is what it is; a check that found nothing new comes back as a single short line instead of the whole output again. Measured through the installed hook: a second check of 5,200 characters came back as about 1,250, and a third check that added nothing came back as 60 — roughly a quarter of the cost for the second look and about one percent for the third, improving the longer the command runs. Nothing is lost, because what is cut is what was already sent. It only shortens a check when the new output genuinely continues the last one seen; anything else passes straight through, so the worst case is a saving that does not happen rather than a wrong answer.
 
 No ambient environment variable documents "this process is running under Copilot CLI" the way Codex/opencode set one, so the shim sets `TOKEN_GOAT_HARNESS_OVERRIDE=copilot_cli` itself before calling `token-goat hook` (same workaround `--pi` uses). Install also writes a token-goat routing block into `~/.copilot/copilot-instructions.md` (the same delimited-block gate written to `~/.claude/CLAUDE.md` and `~/.codex/AGENTS.md`), merged idempotently so any hand-written content outside the markers is preserved byte-for-byte. If you set `COPILOT_HOME`, install follows it — hooks go to `$COPILOT_HOME/hooks/` and the routing block to `$COPILOT_HOME/copilot-instructions.md`, matching where Copilot CLI actually reads them. To install for one project instead of user scope: `token-goat install --copilot --local` (writes `.github/hooks/token-goat.json` and `.github/copilot-instructions.md` in the current project). To remove: `token-goat uninstall --copilot`.
 
@@ -513,6 +515,7 @@ token-goat xlsx-sheets link-map.xlsx
 token-goat xlsx-query link-map.xlsx --sheet Links --columns publication,source,target --head 50
 token-goat pdf-meta manual.pdf
 token-goat pdf-outline manual.pdf
+token-goat pdf-locate manual.pdf "torque spec" --ignore-case
 token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 ```
 
@@ -577,7 +580,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat context-stats [--project <path>]` | Report estimated token overhead from `CLAUDE.md` files and `MEMORY.md` in a project. `--json` for structured output; `--fix` prunes dead-link and duplicate entries from `MEMORY.md` and writes the file (destructive — inspect the report first). |
 | `token-goat bootstrap-audit [--project <path>] [--json]` | Audit Claude Code startup-context contributors without outputting prompt bodies: global/project `CLAUDE.md` totals plus agent/skill frontmatter metadata, largest entries, diagnostics, and CI warning/failure budgets (`--warn-tokens`, `--fail-tokens`, `--warn-bytes`, `--fail-bytes`). |
 | `token-goat memory [--project <path>] [--analyze\|--fix] [--yes]` | Find duplicate/overlapping content across the `CLAUDE.md` files loaded for a project, plus near-duplicate sibling auto-memory files. `--analyze` (default) is report-only. `--fix` removes exact-duplicate lines within a file (the only mechanical, judgment-free fix); duplicate headings and cross-file overlaps are reported as advisory only and never auto-applied. See [Memory analysis and cleanup](#memory-analysis-and-cleanup) below. |
-| `token-goat waste [--project <path>] [--transcript <path>] [--top <n>] [--json]` | Session spend-ledger: parses the current project's Claude Code session transcript and reports token cost by tool, by file, the top N most expensive individual tool calls, files read once and never referenced again, Bash commands run repeatedly without hitting token-goat's own bash-output cache, and the assistant's own text-output cost (generated tokens plus a cache-unaware re-send upper bound). See [Session waste ledger](#session-waste-ledger) below. |
+| `token-goat waste [--project <path>] [--transcript <path>] [--top <n>] [--json] [--copilot]` | Session spend-ledger: parses the current project's Claude Code session transcript and reports token cost by tool, by file, the top N most expensive individual tool calls, files read once and never referenced again, Bash commands run repeatedly without hitting token-goat's own bash-output cache, and the assistant's own text-output cost (generated tokens plus a cache-unaware re-send upper bound). See [Session waste ledger](#session-waste-ledger) below. |
 | `token-goat mcp-audit [--project <path>] [--json]` | MCP server schema cost report: scans .mcp.json for installed MCP servers, estimates per-server token costs from cached tool calls, correlates schema complexity against real call frequency. Outputs as markdown table or JSON. |
 | `token-goat recall ["<query>"] [--type bash\|web\|mcp] [--limit <n>] [--json]` | Full-text search across every cached bash-output, web-output, and mcp-output entry at once — one command instead of remembering which cache type holds a prior result. Ranked by relevance (BM25 via SQLite FTS5). With **no query**, lists every cached entry newest-first instead of searching, so you can browse when the ids have scrolled out of context and you have no term to search for. `--type` narrows to one cache type; `--limit` caps results (default 10). Each hit shows its cache type, id, the exact recall command (`bash-output <id>` / `web-output <id>` / `mcp-output <id>`), and a content snippet. See [Cross-cache recall](#cross-cache-recall) below. |
 | `token-goat hint-stats [--json] [--reset] [--mark-effective <cat>] [--mark-ineffective <cat>]` | Per-category efficacy report for token-goat's discretionary hint hooks: how often each hint category was emitted, how often the agent actually followed its specific suggestion within the next few tool calls, whether the category is currently auto-suppressed, and the bytes each category spent (injected into context) plus an all-time saved/spent/net summary line. `--reset` clears all tracked data; `--mark-effective`/`--mark-ineffective <category>` record a manual vote as a supplement to the automatic signal. See [Hint efficacy tracking](#hint-efficacy-tracking) below. |
@@ -604,6 +607,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat config list / get / set / validate` | Inspect or edit `config.toml` from the CLI. `validate` reports unknown keys with did-you-mean suggestions, plus any project-file or environment value that validation rejected or clamped. A project-root `.token-goat.toml` layers on top of the global config, overriding hint thresholds, indexing settings, etc. for that project only. It may not set the security sections `injection`, `webfetch`, `gdrive`, or `mcp`, nor `indexing.cross_project_symbols`: that file arrives with the repository, so a cloned project could otherwise switch off prompt-injection fencing or empty the fetch allow list for anyone who opened it. Those settings come from the global config or the environment only, and a project file that tries to set one is ignored with a message naming what was dropped. `config get`/`list`/`set` report which layer a value actually resolved from — the project file, an environment variable, or the global config — and where that layer's value was clamped or rejected they say so, naming what was asked for and what is in effect instead. |
 | `token-goat config-get <file> <key>` | Look up one key from a config-shaped file (TOML/INI `key = value`, or YAML) without reading the whole thing. On a `.md` file, a leading `---`-fenced YAML frontmatter block (Jekyll/Hugo/SKILL.md style) is checked first and takes precedence over the TOML/INI fallback; a `.md` file with no frontmatter, or an unclosed fence, falls through to the normal lookup unchanged. |
 | `token-goat pdf-extract <file>` | Extract plain text from a PDF instead of a raw Read. `--pages <spec>` narrows to a page range (e.g. `1-5` or `3`); `--head`/`--tail`/`--grep`/`--max-matches`/`--section` slice the extracted text the same way `bash-output`/`web-output` do. `--layout` heuristically reconstructs column-aware reading order from text-item coordinates instead of raw content-stream order (imperfect on rotated/overlapping text). |
+| `token-goat pdf-locate <file> <pattern>` | Find which pages of a PDF match a regex, with a snippet per match, so you can `pdf-extract --pages` only those pages instead of pulling the whole document. `-i`/`--ignore-case` for case-insensitive matching; `--max-matches <n>` caps how many page matches to collect (default 50); `--context <n>` sets the snippet length around each match (default 80); `--pages <spec>` narrows the scan to a page range; `-j`/`--json` emits `{ file, pattern, matchCount, pages, matches }`. |
 | `token-goat pdf-outline <file>` | List a PDF's bookmark/outline tree with page numbers instead of a raw Read. |
 | `token-goat pdf-meta <file> [--json]` | Page count, title/author, and whether a PDF has an extractable text layer (so you know before extracting whether it's scanned/image-only). `--json` emits `{ pageCount, title, author, hasTextLayer }` — `hasTextLayer` as a real boolean rather than a prose sentence, and an absent title/author as `null` rather than the literal `(none)`. |
 | `token-goat image-meta <file> [--json]` | Dimensions, format, byte size, and what a `shrinkImage` pass would cost — a cheap "should I even look at this" probe that reads `sharp` metadata only and never runs OCR. Requires `sharp`; degrades with a clear message when it's missing. |
@@ -641,7 +645,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat zip-list <archive>` | Entry paths and sizes inside a zip-format archive (`.zip`/`.jar`/`.whl`/`.vsix`/`.nupkg` are all zip containers under the hood) instead of a raw Read or an `unzip -l` shell-out. Reads the central directory only — no member is decompressed just to list it. `--json` for structured output. |
 | `token-goat zip-read <archive> <entry>` | Extract and print exactly one entry's text content from a zip-format archive by its in-archive path, instead of extracting the whole archive to disk. A binary member prints a `[binary content elided by token-goat]` marker instead of raw bytes. |
 | `token-goat pr-slice <pr>` | Surgical GitHub PR reads via `gh` — one file's diff, a single review-comment thread, the description, or CI check statuses, instead of pulling the whole PR payload into context. |
-| `token-goat bridges-status` | Parity matrix of which hooks/commands are wired for each supported harness (Claude Code, Codex, opencode, openclaw, Grok, etc.), side by side. |
+| `token-goat bridges-status` | Parity matrix of which hooks/commands are wired for each supported harness (Claude Code, Codex, opencode, openclaw, Grok, etc.), side by side. A `verified` column says how each row was established — `dogfooded` (driven against the real harness binary), `sourced` (read out of the harness's own source or declarations), or `documented` (from its docs only) — so a claim built from reading alone is never presented as one that was tested. |
 | `token-goat commands` | Machine-readable manifest of every registered command, its description, options, and arguments (including subcommands like `worker start`). `--json` emits it as structured JSON for external tooling (shell completion, doc generators, scripts) instead of the default text listing. `--grep PATTERN` narrows the manifest to commands whose name, description, or aliases match; a parent command that matches keeps all its subcommands, a parent that only has a matching child keeps just that child; no matches prints `no matches` and exits 0. |
 | `token-goat mcp-serve` | Run token-goat as an MCP stdio server exposing all 18 tools: read/symbol/section/outline/skeleton/semantic/index_status/refs/brief/map/changed/grep/imports/exports/compress_text/retrieve_text/handoff_create/handoff_resolve. |
 | `token-goat version` | Print the token-goat version. |
@@ -654,7 +658,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat project exclude <path>` | Add a project root to the blocklist so the worker never indexes it. Writes the resolved absolute path to `[worker] blocked_roots` in `config.toml`; idempotent. It also removes anything already indexed under that path and says how many files went, so excluding a directory means its contents stop being readable through `symbol` rather than merely stopping future indexing. Remove the entry from the config to re-enable indexing, then run `token-goat index` to bring the contents back. |
 | `token-goat project prune [--dry-run]` | Remove blocked/excluded roots that no longer exist on disk. `--dry-run` previews removals without touching the config file. Useful after deleting or moving projects. |
 | `token-goat install` | Wire up hooks (and, with the harness flags below, other AI tool integrations). No `--dry-run` or `--verify` flag — run `token-goat doctor` after install to audit the result. |
-| `token-goat doctor` | Confirm everything is wired correctly. Surfaces install state, cold-import timing, cache hit rates, compaction-budget telemetry, opt-in flag status, and canonical-root sanity. A **Security** section reports the posture in one place: whether offline mode is on, whether injection scanning is on, whether the Google Drive integration is enabled, whether fetching runs against an allow list or a deny list, whether MCP reads are confined to the project root, and whether the data directory is readable by other local users. It only warns when a protection that ships on has been switched off, so a default install stays quiet. It read-only audits `~/.copilot/mcp-config.json` for globally configured Chrome DevTools or Playwright `npx` launchers, recommending project scope or removal when inactive; it never prints server configuration or secrets. On Windows, it also reports duplicate Chrome DevTools/Playwright MCP launchers and orphaned Node processes without terminating anything. Pass `--context` to show the **Context footprint** section: a fill bar with severity (ok / warn / high / URGENT), per-component breakdown (skills catalog, loaded skill bodies, CLAUDE.md+MEMORY.md, conversation estimate), session-to-session growth trend with sessions-to-URGENT projection, and tiered compaction recommendations (Tier 0–4) naming the exact commands to run. Auto-shown when fill > 40 % or any loaded skill > 2 K tokens lacks a compact. `--json` emits the check results (one entry per check, with `ok`/`warn`/`fail` status) as JSON instead of text. |
+| `token-goat doctor` | Confirm everything is wired correctly. Surfaces install state, cold-import timing, cache hit rates, compaction-budget telemetry, opt-in flag status, and canonical-root sanity. A **Tool names** check reports any tool name a harness sent that reached no handler wanting it, and calls out the ones that differ from a handled name only by capitalisation or punctuation — the signature of a bridge that forgot to rename something, which is otherwise invisible. A **Security** section reports the posture in one place: whether offline mode is on, whether injection scanning is on, whether the Google Drive integration is enabled, whether fetching runs against an allow list or a deny list, whether MCP reads are confined to the project root, and whether the data directory is readable by other local users. It only warns when a protection that ships on has been switched off, so a default install stays quiet. It read-only audits `~/.copilot/mcp-config.json` for globally configured Chrome DevTools or Playwright `npx` launchers, recommending project scope or removal when inactive; it never prints server configuration or secrets. On Windows, it also reports duplicate Chrome DevTools/Playwright MCP launchers and orphaned Node processes without terminating anything. Pass `--context` to show the **Context footprint** section: a fill bar with severity (ok / warn / high / URGENT), per-component breakdown (skills catalog, loaded skill bodies, CLAUDE.md+MEMORY.md, conversation estimate), session-to-session growth trend with sessions-to-URGENT projection, and tiered compaction recommendations (Tier 0–4) naming the exact commands to run. Auto-shown when fill > 40 % or any loaded skill > 2 K tokens lacks a compact. `--json` emits the check results (one entry per check, with `ok`/`warn`/`fail` status) as JSON instead of text. |
 | `token-goat baseline` | Emit a project map: file count, per-language file counts, the top indexed symbols (by name/kind/location), and the most recently modified files. `--subagent` emits a terser variant (fewer symbols, fewer recent files) for context handed to a freshly spawned subagent; `--json` for the machine-readable form. |
 | `token-goat compact-doc <path>` | Build an extractive compact sidecar for a large reference doc (`.md`/`.markdown`). The compact is stored in the token-goat data dir as a SHA-keyed sidecar; `pre_read` serves it in place of the full file when it exists and is fresh, saving 80–95% of context tokens. Use `--force` to rebuild, `--sentences N` to control lines per section (default 2), `--show` to print the result. The sidecar is automatically marked stale when you edit the source file. Config: `[hints] stable_doc_compacts = true` (default on). |
 
@@ -736,6 +740,24 @@ Total tokens: 18420
 
 `--top <n>` controls how many entries appear under "Top expensive tool calls" (default 10). `--json` prints the same report as machine-readable JSON instead.
 
+`--copilot` reads a GitHub Copilot CLI session instead, from `<copilot-home>/session-state/<id>/events.jsonl`. It is a different report rather than the same one with different inputs, because Copilot writes down its own token accounting at shutdown and token-goat reports those numbers rather than estimating them:
+
+```
+$ token-goat waste --copilot
+
+## Per-request fixed overhead (Copilot's own token counts)
+  System prompt:     8,981 tok
+  Tool definitions:  11,548 tok
+  Conversation:      722 tok
+
+## Tool definitions by MCP server (estimated)
+  github-mcp-server: 6 tools, 6 KB, ~2,135 tok
+  ~2,135 tok estimated across 1 server, re-sent every request.
+  Copilot counted 11,548 tok of tool definitions in total, so this is roughly 18.5% of it.
+```
+
+The fixed overhead is the largest number in a Copilot session and no hook can reach it: Copilot assembles the system prompt and the tool definitions natively, with nothing between assembly and send. Only configuration moves it. The per-server breakdown exists to make that configuration decision possible, since one aggregate says the tool definitions are expensive without saying which tools. It is read from Copilot's own MCP tool cache, counts only the fields a model is actually sent, and is labelled an estimate throughout: it comes from byte length rather than Copilot's tokeniser, and it deliberately does not add up to Copilot's total, because Copilot's own built-in tools are not cached there.
+
 The "Assistant output" section is separate from the tool-call ledger above it: `generatedTokens` is what was actually paid, once, to produce the assistant's own text turns. `resendCeilingTokens` is a cache-unaware upper bound on how much re-sending those turns as conversation history on every later request could cost — not real spend, since Claude Code's prompt caching bills a repeated conversation prefix at cache-read rates, a fraction of full input price. Treat it as a ceiling on how bad unbounded verbosity could get, not as a dollar figure.
 
 ### Cross-cache recall
@@ -1225,7 +1247,7 @@ Outbound network is reserved to these explicit cases:
 - Google Drive API calls, only if you already authorized Drive in Claude Code. Token-goat never prompts for its own auth.
 - Image fetches from URLs: either explicit via `token-goat fetch-image <url>`, or when the AI agent issues a WebFetch call that returns image content — the hook intercepts and shrinks the image. The URL always originates from the agent's work, not from token-goat itself.
 - `token-goat screenshot <url>` navigates a headless browser to the URL you give it, subject to the target restrictions described below.
-- The first `token-goat semantic` run on a machine downloads the embedding model from `huggingface.co`, pinned to an immutable commit rather than a mutable branch. Subsequent runs use the local cache and make no network call. Skip the download entirely by setting `indexing.embeddings_enabled = false` (it is on by default), in which case `semantic` falls back to full-text search.
+- The first `token-goat semantic` run on a machine downloads the embedding model from `huggingface.co`, pinned to an immutable commit rather than a mutable branch and checked against a recorded SHA-256 and byte length before it is used, and only once `onnxruntime-node` has been installed (see below — it is not part of a default install). Subsequent runs use the local cache, re-verify it, and make no network call. Skip the download entirely by setting `indexing.embeddings_enabled = false` (it is on by default), in which case `semantic` falls back to full-text search.
 - The first optical-character read of an image downloads the English language data (about 4 MB) from `cdn.jsdelivr.net`, at a fixed version path. Subsequent reads use the local cache. This happens for an explicit `token-goat image-text`, and also for the automatic text extraction the image-shrink hook performs when the agent reads a screenshot; turn the automatic one off with `image_shrink.ocr_enabled = false`.
 
 **One switch for all of it.** Set `network.offline = true` (env `TOKEN_GOAT_OFFLINE`) and every one of the paths above refuses instead of connecting, saying so rather than failing quietly. Anything already cached keeps working: a machine that has the embedding model still runs `semantic`, and one that has the language data still reads text out of images. This is one of the settings a per-project config file may not touch, so cloning a repository cannot switch it back off.
@@ -1234,7 +1256,13 @@ Outbound network is reserved to these explicit cases:
 
 **Security reports.** See [SECURITY.md](SECURITY.md). Email `token-goat@dfkhelper.com`; do not file as a GitHub issue. Reports are acknowledged within 7 days; coordinated disclosure with a 90-day default window.
 
-**Dependency advisories.** `npm audit` on the published package is not empty. The residual findings all trace to three packages that have no forward patch, two of which are optional and can be left out with `npm install --omit=optional`. Each one is named, with why it does or does not reach you, under [Dependency advisories](SECURITY.md#dependency-advisories).
+**Dependency advisories.** `npm audit` on the published package is empty, for a default install as well as for `npm install --omit=optional`. That took removing the package the findings all came through: `@xenova/transformers`, which supplied the embedding half of `semantic` and carried a critical `protobufjs` advisory plus five more that had no forward patch. It is gone entirely now — the tokenizer and the ONNX runner are token-goat's own code over `onnxruntime-node`, which is 17 packages where the old one was 80. That runtime is opt-in and not installed by default. `semantic` still works without it, on keyword search. Both `semantic` itself and `token-goat doctor` say so, and print the one command that brings the embeddings back:
+
+```bash
+npm install -g onnxruntime-node   # drop -g if token-goat is a project dependency
+```
+
+That command is the one thing here that is not clean: `onnxruntime-node` pulls an `adm-zip` below 0.6.0, which carries [one high advisory](https://github.com/advisories/GHSA-xcpc-8h2w-3j85) that npm reports twice. It is reachable only from that package's own install script, unpacking the binary it just downloaded. The full accounting — including the override that clears it, why co-installing a fixed `adm-zip` does not, and what the old package cost — is under [Dependency advisories](SECURITY.md#dependency-advisories).
 
 **Verifying what you installed.** Every published version is built and pushed by one pinned workflow when a GitHub release is published, with npm provenance, so `npm audit signatures` verifies the tarball against the commit that produced it. Details in [Verifying what you installed](SECURITY.md#verifying-what-you-installed).
 

package/SECURITY.md

@@ -41,29 +41,59 @@ The following are not treated as security issues unless paired with a working pr
 
 ## Dependency advisories
 
-`npm audit` gives three different answers for Token-Goat depending on what you scan, and the gap between them is the first thing to understand. All three numbers below are reproducible with the commands shown.
+`npm audit` reports Token-Goat clean whichever way you scan it, and that has only been true of a default install since the embedding model became opt-in. All three numbers below are reproducible with the commands shown. The one thing that is not clean is the opt-in package itself, and it is described further down rather than folded into these rows.
 
 | What you scan | Command | Result |
 | --- | --- | --- |
 | this repository | `npm audit` | clean, development dependencies included |
-| an install without optional packages | `npm install --omit=optional token-goat` then `npm audit --omit=dev --omit=optional` | clean, 40 packages |
-| a default install | `npm install token-goat` then `npm audit --omit=dev` | the optional packages in the table below |
+| an install without optional packages | `npm install --omit=optional token-goat` then `npm audit --omit=dev --omit=optional` | clean, 2 packages |
+| a default install | `npm install token-goat` then `npm audit --omit=dev` | clean, 70 packages |
 
-The repository is clean because [`package.json`](package.json) carries an `overrides` block that pins five transitive packages to patched versions. **npm applies `overrides` only in the root project**, so those pins do not travel to anyone who installs Token-Goat as a dependency. We are saying so plainly rather than letting a clean repository scan stand in for a clean install: if your scanner reads this repository or its lockfile it will report nothing, and that is not the whole picture.
+The repository carries an `overrides` block in [`package.json`](package.json) that pins six transitive packages to patched versions, and **npm applies `overrides` only in the root project**, so those pins do not travel to anyone who installs Token-Goat as a dependency. That distinction used to matter a great deal, because a clean repository scan was standing in for an install that was not clean. It decides one thing now, and the section on the embedding runtime below says which. Otherwise the packages those pins were protecting against are not in a consumer's tree at all, whether or not the pins travel; the block stays because the development tree still resolves them.
 
-Without the overrides a consumer resolves `protobufjs` at 6.x, which carries more advisories than the 7.x line the repository would otherwise use. Every one of those paths is optional.
+There is no table of reaching advisories here any more, and the last entry in it is worth recording rather than deleting. `@xenova/transformers` was an optional dependency, which in npm means everyone got it unless they asked not to, and it carried [`protobufjs`](https://github.com/advisories/GHSA-xq3m-2v4x-88gg) through `onnx-proto` and `onnxruntime-web` -- one critical advisory and four more -- plus its own nested, older `sharp` carrying [four inherited libvips CVEs](https://github.com/advisories/GHSA-f88m-g3jw-g9cj). None of the six was fixable from here: every one sits in a transitive package with no patched version, and the pins that would have solved it do not travel to consumers.
 
-| Package | Advisories it carries | Where it loads | Why the advisory does not reach you through Token-Goat |
-| --- | --- | --- | --- |
-| `@xenova/transformers` | [`protobufjs`](https://github.com/advisories/GHSA-xq3m-2v4x-88gg) (critical), `onnx-proto`, `onnxruntime-web`, and its own pinned `sharp` | optional; loaded only when semantic search builds or queries embeddings | not mitigated, so it is the one to weigh. Skip it with `npm install --omit=optional`, or leave `indexing.embeddings_enabled` off, and the code never loads |
+That package is gone entirely now. It had shipped no release since May 2024, so none of those six was going to be fixed upstream either, and what it did for Token-Goat was three separable things: tokenize the text, run an ONNX graph, average the result. All three are in [`src/embed_tokenizer.ts`](src/embed_tokenizer.ts) and [`src/embed_model.ts`](src/embed_model.ts) now, over [`onnxruntime-node`](https://www.npmjs.com/package/onnxruntime-node) -- ONNX Runtime's own Node binding, from the same project that produces the format the model is published in. Measured standalone into an empty project, that is 17 packages where `@xenova/transformers` was 80.
+
+The embedding runtime is opt-in, the same way the model was. `semantic` still works without it, because that command always consults keyword search alongside the vectors, so nothing errors and nothing comes back empty; what goes away is the embedding half, and with it the ability to match on meaning rather than words. To get it back:
+
+```bash
+npm install -g onnxruntime-node   # drop -g if token-goat is a project dependency
+```
+
+**That command is the one thing on this page that is not clean, and the number is two.** `onnxruntime-node` depends on [`adm-zip`](https://www.npmjs.com/package/adm-zip) below 0.6.0, which carries [GHSA-xcpc-8h2w-3j85](https://github.com/advisories/GHSA-xcpc-8h2w-3j85) -- a crafted ZIP file provoking a 4 GB allocation -- and npm reports it twice, once against `adm-zip` and once against `onnxruntime-node` for depending on it. Both are the same finding. It is reachable only from `onnxruntime-node`'s own postinstall script, which uses it to unpack the prebuilt binary it has just downloaded from Microsoft; nothing in Token-Goat passes it an archive, and no file a user supplies reaches it. This repository pins `adm-zip` past it in `overrides`, which is why the repository row above is clean, and by the rule stated above that pin does not travel: run the command and you will resolve the 0.5 line and see those two. If that matters where you are, the fix is an `overrides` entry of your own:
+
+```json
+{ "overrides": { "adm-zip": "^0.6.0" } }
+```
+
+in the package.json of the project you install into, then reinstall. Measured, because the obvious alternative does not work: installing `adm-zip@^0.6.0` alongside `onnxruntime-node` changes nothing, since the version it asks for is `^0.5.16` and npm nests a second copy rather than sharing yours. An override is what actually replaces it, and the prebuilt binary still extracts afterwards -- that pairing is what this repository builds and tests on. A global `npm install -g onnxruntime-node` has no project manifest to put an override in, so there it stands.
+
+Nothing else is needed. The index notices on its own: files skipped while the model was absent are recorded as skipped for that reason specifically, so the next index pass re-embeds them rather than treating them as already done. `token-goat doctor` reports which of the three states you are in -- available, switched off in config, or not installed with the command above -- because a fallback that works silently is exactly the kind of thing nobody discovers on their own.
+
+The model weights are no longer a package at all. They are downloaded once, on first use, from a URL whose every component is a constant in [`src/embed_model.ts`](src/embed_model.ts) -- repository, revision and filename alike, with nothing caller-supplied anywhere in it. The revision is an immutable commit rather than a branch, and each file is checked against a recorded SHA-256 and an exact byte length, on download and again on every load, with the length enforced while the body is still streaming so an overrunning response is cut off rather than written out. The digest is what is trusted, not the hostname: `huggingface.co` accepts uploads from anyone, which is the shape of [CVE-2026-54316](https://github.com/advisories/GHSA-fg94-h982-f3mm), so allowlisting the host would decide nothing. Because the digest decides, following the redirect that `resolve` issues to Hugging Face's CDN is safe. With `network.offline` set, the download is refused and the path says so rather than degrading quietly; files copied into the cache directory by hand are still accepted, because they are checked the same way.
+
+The old model had cost something before any of this, through the nested `sharp` it carried. That `sharp` ships its own libvips binaries, and loading the model eagerly put them ahead of Token-Goat's own `sharp` in the Windows DLL search order, which broke image shrinking with `ERR_DLOPEN_FAILED` while `sharp` loaded perfectly well on its own. `onnxruntime-node` carries no `sharp` and no libvips, so that particular collision cannot recur -- but the load is still deferred until something actually asks to embed, in [`src/embed_model.ts`](src/embed_model.ts), because it is a native addon whose DLLs have no business being loaded into every hook invocation that never embeds anything.
+
+`npm install --omit=optional` remains available and gives a smaller install still, at 2 packages -- Token-Goat and [`jsonc-parser`](https://www.npmjs.com/package/jsonc-parser), and nothing else. Every command starts either way; the ones that need a package you skipped say so. The `xlsx-*`, `docx-*` and `pptx-*` commands report that fflate is not installed rather than failing oddly, and `zip-list`/`zip-read` do the same.
+
+`exceljs` used to appear in that table, carrying [`uuid`](https://github.com/advisories/GHSA-w5hq-g745-h8pq). It is now a development dependency instead. The `xlsx-*` commands read the workbook container directly, the same way the `.docx` and `.pptx` readers already did, so `exceljs` is only a test fixture writer now. That removes 55 packages from a default install, including every deprecated one in the tree.
+
+`fast-xml-parser` has moved the same way, and for a reason worth stating even though it carries no advisory. Its 5.x line splits what used to be one transitive package into six ([`@nodable/entities`](https://www.npmjs.com/package/@nodable/entities), [`fast-xml-builder`](https://www.npmjs.com/package/fast-xml-builder), [`is-unsafe`](https://www.npmjs.com/package/is-unsafe), [`path-expression-matcher`](https://www.npmjs.com/package/path-expression-matcher), [`xml-naming`](https://www.npmjs.com/package/xml-naming) and `strnum`, which itself now pulls [`anynum`](https://www.npmjs.com/package/anynum)). Five new maintainer surfaces arrived inside a version range an existing install accepts on its own, without anyone deciding to take them on, and that is the shape of supply-chain exposure regardless of whether any one of those packages is doing anything wrong today. What we used was one constructor with four options over machine-generated XML, so it is now [`src/xml_parser.ts`](src/xml_parser.ts) instead, with nothing underneath it. That parser reads no DTD and supports no entity declarations at all, so XXE and entity-expansion attacks are closed by construction rather than by a limit. `fast-xml-parser` stays as a development dependency, where a differential test holds the local parser to its exact output.
+
+`@modelcontextprotocol/sdk` has moved to a development dependency as well, and it is the largest single reduction of the three. The SDK is Anthropic-maintained and good at what it does, but it is built to be every MCP participant at once -- client and server, stdio and HTTP and SSE, OAuth, resources, prompts, sampling, tasks -- and it charges every install for all of it. It brought 99 packages, among them two HTTP frameworks ([`express`](https://www.npmjs.com/package/express) and [`hono`](https://www.npmjs.com/package/hono)), [`cors`](https://www.npmjs.com/package/cors), [`body-parser`](https://www.npmjs.com/package/body-parser), [`express-rate-limit`](https://www.npmjs.com/package/express-rate-limit), [`jose`](https://www.npmjs.com/package/jose), [`pkce-challenge`](https://www.npmjs.com/package/pkce-challenge), [`qs`](https://www.npmjs.com/package/qs) and [`path-to-regexp`](https://www.npmjs.com/package/path-to-regexp). Token-goat runs one stdio server that registers tools and answers `tools/call`, so no HTTP transport is reachable from it and none of those packages can execute here; they were install weight and attack surface and nothing else. What token-goat used of the SDK was five calls, and that is now [`src/mcp_jsonrpc.ts`](src/mcp_jsonrpc.ts) and [`src/mcp_stdio.ts`](src/mcp_stdio.ts). The SDK stays as a development dependency, and that is load-bearing rather than incidental: every MCP test in the repository drives our server through the SDK's own client over its own transport, so the reference implementation checks our wire format on every run. A default install drops from 250 packages to 151, and to 106 once the embedding model comes out too.
+
+`better-sqlite3` is gone too, and it was the last large thing left. It is a native addon, which is what made it expensive in every way this page cares about: 36 packages, an install script, a compile step, and the only package left in a consumer's tree still marked deprecated. Node ships the same SQLite engine in core. [`node:sqlite`](https://nodejs.org/docs/latest-v22.x/api/sqlite.html) stopped requiring a flag in Node 22.13.0, and the handful of API differences that matter -- it has no `pragma()`, no `transaction()`, and no `reader` flag on a prepared statement -- are covered by [`src/sqlite_driver.ts`](src/sqlite_driver.ts), which presents the same shape the rest of the code already called. Nothing else changed: the same engine, the same file format, the same WAL and busy-timeout settings, `sqlite-vec` still loading as an extension. The floor moves to Node 22.16.0, which is where `statement.columns()` and `database.isTransaction` landed.
+
+`better-sqlite3` stays as a development dependency, and like the MCP SDK that is load-bearing. Three parts of the driver are reimplementations rather than passthroughs, and a test written only against the driver would pin whatever the driver does, bugs included. So [`tests/sqlite_driver.test.ts`](tests/sqlite_driver.test.ts) runs both libraries over the same input and requires the answers to match -- most carefully for the `reader` flag, which is the third defence-in-depth layer in the `sqlite-query` read-only guard and the one thing the driver derives rather than reads. The two libraries differ in exactly one place, deliberately: asked for an integer above 2^53 without big-integer reads switched on, `better-sqlite3` returns the nearest double and `node:sqlite` refuses. Silently rounding is the behaviour `sqlite-query` already had to defend against, so the refusal is kept, and that test pins the difference rather than papering over it. That takes a default install from 106 packages to 70, and an install without optional packages from 40 to 2.
 
-`npm install --omit=optional` gives you an install without it, and `npm audit` reports it clean. Every command still starts; the ones that need a package you skipped say so. The `xlsx-*`, `docx-*` and `pptx-*` commands report that fflate is not installed rather than failing oddly, `zip-list`/`zip-read` do the same, and `semantic` keeps working on keyword search alone: it is the embedding half that goes away, not the command.
+Every package count on this page is a measurement, not a constant, and it was taken the same way each time: `npm install <the package>` into an empty project, then counting the directories under `node_modules`, the package itself included. They are counts as of 2026-08-22 and they drift upward on their own, because the version ranges a dependency declares resolve to whatever is newest at install time and other people's trees grow. This paragraph said 238 and 87 for a while for exactly that reason: both were true when they were written and neither was true a few releases later. Re-measure before quoting them rather than assuming they still hold. The default figure additionally depends on where you stand: `sharp`, `@napi-rs/canvas` and `sqlite-vec` each publish a prebuilt binary per platform, an install takes only the one that matches, and the rest are skipped, so a default install is a few packages larger on Linux than on Windows or macOS. The 70 above was measured on Windows x64, and it is the largest of the three. The no-optional install has no prebuilt binary in it at all, so 2 is 2 everywhere.
 
-`exceljs` used to appear in that table, carrying [`uuid`](https://github.com/advisories/GHSA-w5hq-g745-h8pq). It is now a development dependency instead. The `xlsx-*` commands read the workbook container directly with `fflate` and `fast-xml-parser`, the same two packages the `.docx` and `.pptx` readers already used, so `exceljs` is only a test fixture writer now. That removes 55 packages from a default install, including every deprecated one in the tree.
+One half of that drift is checked rather than trusted. [`tests/guards/dependency_advisory_disclosure.test.ts`](tests/guards/dependency_advisory_disclosure.test.ts) resolves both installs out of [`package-lock.json`](package-lock.json) the way npm resolves them, platform gating included, and holds this page to the answer: the no-optional figure has to match exactly, and the default figure may not be smaller than the lock file already proves it must be. That catches every package this project adds or removes on its own. It cannot catch the other half, which is other people's trees growing inside ranges an install already accepts, and that is what the date is for.
 
 `html-to-text` used to appear in that table, carrying [`deepmerge-ts`](https://github.com/advisories/GHSA-ggr8-5vv4-36mx). It is now a development dependency instead. esbuild inlines it into `dist/token-goat.mjs` at build time and nothing in the published bundle imports it, so it was a runtime dependency in name only: moving it removes `html-to-text`, `deepmerge-ts`, `htmlparser2`, `selderee`, and `dom-serializer` from an installed copy while the HTML-to-text output stays byte-for-byte identical. That is what takes the no-optional install to zero, and it is better than the alternative we had considered, rolling `html-to-text` back to 9.x: that version pins `htmlparser2` two majors lower, and `htmlparser2` is what parses fetched pages, so it would have traded an advisory in an options merger for an older parser on the one path that handles untrusted input.
 
-Five more packages were in `dependencies` for the same reason and have moved the same way: `commander`, `csv-parse`, `js-yaml`, `smol-toml` and `zod`. esbuild inlines each one into the bundle, and the published bundle resolves none of them, so a consumer was downloading code the artifact already carried. Moving them takes an install without optional packages from 46 packages to 40. `better-sqlite3` and `jsonc-parser` stay, because the bundle really does load them at run time: the first is a native addon, and the second is reached through `createRequire` rather than an import esbuild can inline. `zod` is the largest of the five on disk and the one that saves least in practice, because `@modelcontextprotocol/sdk` and `puppeteer-core` both depend on it and a default install still gets it from them.
+Five more packages were in `dependencies` for the same reason and have moved the same way: `commander`, `csv-parse`, `js-yaml`, `smol-toml` and `zod`. esbuild inlines each one into the bundle, and the published bundle resolves none of them, so a consumer was downloading code the artifact already carried. Moving them takes an install without optional packages from 46 packages to 40. `jsonc-parser` stays, because the bundle really does load it at run time: it is reached through `createRequire` rather than an import esbuild can inline. `zod` is the largest of the five on disk, and when that move was made it saved least in practice, because `@modelcontextprotocol/sdk` and `puppeteer-core` both depended on it and a default install still got it from them. Neither is true any more: the puppeteer-core major bump dropped its `zod` dependency, and the MCP SDK is a development dependency now for the reason below. A default install carries no copy of `zod` at all.
 
 Direct dependencies with a forward patch are kept current rather than pinned: `sharp` and `puppeteer-core` were both moved across a major version to clear their advisories.
 
@@ -96,15 +126,14 @@ the ones this machine installed, 21 entries need a human answer: 7 declare a lic
 cannot resolve, and 14 carry a copyleft term. All 21 arrive through optional dependencies. Install
 with `npm install --omit=optional token-goat` and not one of them is present.
 
-**Declarations a scanner cannot resolve.** Both are upstream mistakes, and both are the same
-mistake this project made in its own manifest and fixed. `SEE LICENSE IN <file>` is npm's form for
-a license that is not on the SPDX list, and `MIT OR Apache` is not a valid expression because the
-identifier is `Apache-2.0`. For the second, `npm sbom` emits no `licenses` field at all rather than
-an unresolvable one.
+**Declarations a scanner cannot resolve.** One remains, and it is an upstream mistake of the same
+kind this project made in its own manifest and fixed: `MIT OR Apache` is not a valid expression,
+because the identifier is `Apache-2.0`. `npm sbom` emits no `licenses` field at all for it rather
+than an unresolvable one. (`flatbuffers` was listed here too, declaring `SEE LICENSE IN LICENSE.txt`
+while actually granting Apache-2.0. It arrived through `@xenova/transformers` and left with it.)
 
 | Package | Declares | Actually grants | Reached through |
 | --- | --- | --- | --- |
-| `flatbuffers` | `SEE LICENSE IN LICENSE.txt` | Apache-2.0, in its own `LICENSE.txt` | `@xenova/transformers` |
 | `sqlite-vec` and its 5 platform packages | `MIT OR Apache` | MIT or Apache-2.0, your choice | direct optional dependency |
 
 **Copyleft terms.** Two families, and neither puts a copyleft obligation on Token-Goat's own code.