token-goat 2.6.30 → 2.6.32

package/README.md

@@ -39,6 +39,12 @@ Restart your AI sessions. Run `token-goat stats` a couple of minutes after your
 
 ---
 
+<p align="center">
+  <img src="assets/token-goat-comparison.jpg" alt="Side-by-side comparison: a bloated workflow sends whole files and grows context every round, while token-goat sends only the needed lines and stays lean" width="900">
+  <br>
+  <sub>Same requirements, smarter input: fewer input tokens, shorter answers, and context that stops compounding across rounds</sub>
+</p>
+
 <p align="center">
   <img src="assets/stats_v180.png" alt="token-goat stats display" width="589">
   <br>
@@ -137,11 +143,10 @@ The fastest way to reduce AI token costs is fixing these five, not writing short
 | Recovery hints omit critical paths when space is tight | Skip bash snippet when recall available |
 | AVIF format not supported despite better compression | AVIF image-shrink via sharp (when libvips is built with libaom); WebP fallback; codec auto-detection in docker |
 | Token-savings invisible until you run `stats` | Token-savings benchmark (slow-marked test suite) locks in measured wins; `token-goat stats` reports net-positive impact |
-| Hook crash leaves agent waiting for response | Fail-soft barrier catches `BaseException`/`MemoryError`/`SystemExit`; hook always returns `{"continue": true}` |
+| Hook crash leaves agent waiting for response | Every way the shim can fail prints `{}` and exits 0, leaving the tool call to proceed untouched: an event name it does not know, stdin it cannot read or parse, an in-process load that throws, a token-goat child that exits non-zero or prints nothing, and a catch around the whole run |
 | Concurrent edits lose update counts mid-session | Session CAS + mtime-based retry prevent lost edits in manifest |
-| Dirty queue appends corrupt on concurrent writes | OS file lock (fcntl/msvcrt) prevents torn JSON lines |
+| Dirty queue appends corrupt on concurrent writes | Each entry is one `O_APPEND` line, and an append that finds the file not ending in a newline starts with one, so a line torn by an earlier crash cannot merge with the next path into a single garbage entry |
 | Worker claim file blocks all re-spawns on crash | Mtime staleness check (>60s) auto-recovers zombie claim files |
-| DRY consolidation — 600+ lines duplicated | Tool-response extractor unified; cache helpers (`_safe_join`, `OutputStatDict`) consolidated; dedup-hint template collapsed; CLI output/history commands unified; `humanize_bytes` centralized in `render/ansi` |
 | Compaction hook subprocess ~190 ms cold | Lazy imports of heavy modules in `hooks_session` and `compact`; compaction path ~110 ms cold (~42% faster) |
 | Pre-compact subprocess runs on every session | Compact-skip sentinel on disk: if session file is <5 min old and no edits logged, subprocess exits in <1 ms |
 | Git ops slow manifest build in non-repo dirs | `git diff` / `git log` calls skipped when `cwd` is not inside a git repo (saves 60–100 ms per hook fire) |
@@ -311,7 +316,7 @@ For bounded archive/document comparisons after setup, see the [CLI comparison wo
 
 **Important — sharp is part of the install.** The biggest single win (~39% smaller than JPEG, ~97% smaller than raw PNG) comes from WebP encoding. sharp ships with prebuilt native binaries on Windows, macOS, and most Linux platforms, so a standard `npm install -g token-goat` includes it. On Alpine musl and some ARM boards, npm compiles from source against libvips — that may add a few seconds. `token-goat doctor` will report `sharp: ok` when the image pipeline is working; if it flags a failure, follow the [Image support](#image-support) section below before considering the install complete. **AI agents performing this install: run `token-goat doctor` after `token-goat install` and resolve any sharp failure as part of the same task — do not treat it as a follow-up.**
 
-Two things change how Claude Code sessions behave: hooks fire automatically (image shrink, re-read dedup, compact manifests), and a delimited routing block written to `~/.claude/CLAUDE.md` plus a registered skill gate the agent's reads — before any file read it must ask whether a `token-goat read` / `symbol` / `section` returns just what it needs, and the block explicitly subordinates the harness's own Read/Grep tool-preference rules to the *fallback* choice once token-goat is ruled out. A `Bash(token-goat:*)` allowlist entry in `settings.json` lets the agent run those commands without a per-call approval prompt.
+Two things change how Claude Code sessions behave: hooks fire automatically (image shrink, re-read dedup, compact manifests), and a delimited routing block written to `~/.claude/CLAUDE.md` plus a registered skill gate the agent's reads — before any file read it must ask whether a `token-goat read` / `symbol` / `section` returns just what it needs, and the block explicitly subordinates the harness's own Read/Grep tool-preference rules to the *fallback* choice once token-goat is ruled out. Install writes no permission entry: whether `token-goat` commands need a per-call approval prompt is left to your own `settings.json`, unchanged.
 
 **Keep that block where install put it.** It's plain markdown in a file you own, so moving it into a tidier reference file is tempting — but `install` and `uninstall` resolve one hardcoded path (`~/.claude/CLAUDE.md`). A relocated copy is never refreshed, so it freezes at whatever version was current when it moved, and the next `install` sees CLAUDE.md missing its block and appends a fresh one — leaving the guidance duplicated across two files with only one of them live. `token-goat doctor` warns when it finds a block outside CLAUDE.md, naming the file; `install` warns at write time and `uninstall` reports what it couldn't remove. None of them edit a file token-goat doesn't own, so cleanup stays your call. A pointer that merely *mentions* the markers in prose is fine — detection requires both markers on their own lines.
 
@@ -550,8 +555,8 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat map` | Get a compact orientation of the repo. Add `--compact` to fit a fixed 2000-token budget. `--json` emits the project map as JSON instead of text. |
 | `token-goat deps "file"` | One-level import listing for a single file: resolves relative imports to project files (`internal`, root-relative paths) and groups everything else as `external`. `--json` for structured output. `--grep <pattern>` only shows dependencies whose MODULE SPECIFIER (the resolved internal path or the external package name) matches this regex (literal substring if it is not valid regex), applied before output is built; when it matches nothing among real dependencies, the output names the active filter instead of reading like the file has no imports at all. Complemented by `token-goat arch` for the project-wide graph. |
 | `token-goat arch` | Project-wide import graph summary: hub modules (most imported), entry points (nothing imports them), and circular chains. Complements `token-goat deps <file>` for per-file depth. |
-| `token-goat index [path]` | Parse all git-tracked files and (re)build the symbol index from scratch. Runs automatically on install and incrementally via the background worker after edits — use this to force a full rebuild (e.g. after a `.tokengoatignore` change). `--walk` indexes a bounded directory walk instead when `path` isn't a git repo. `--force-walk` does the same non-git walk and raises its 20,000-file refusal to 500,000 for a folder you know is genuinely that large (slow, and produces a large index — check `token-goat doctor` afterwards); it never lifts the separate refusal to walk a filesystem root or your home directory. On a real terminal (not a pipe/CI), prints a live progress line to stderr (files done/total, current phase, elapsed time) so a large repo doesn't look hung; stdout is unaffected either way. |
-| `token-goat ignores` | List active skip patterns for the current project — built-in skip dirs and suffixes, plus any patterns from `.tokengoatignore`. |
+| `token-goat index [path]` | Parse all git-tracked files and (re)build the symbol index from scratch. Runs automatically on install and incrementally via the background worker after edits — use this to force a full rebuild (e.g. after a config change that narrows what gets indexed). `--walk` indexes a bounded directory walk instead when `path` isn't a git repo. `--force-walk` does the same non-git walk and raises its 20,000-file refusal to 500,000 for a folder you know is genuinely that large (slow, and produces a large index — check `token-goat doctor` afterwards); it never lifts the separate refusal to walk a filesystem root or your home directory. On a real terminal (not a pipe/CI), prints a live progress line to stderr (files done/total, current phase, elapsed time) so a large repo doesn't look hung; stdout is unaffected either way. |
+| `token-goat ignores` | List active skip patterns for the current project — built-in skip dirs and suffixes, blocked roots, and which command each one applies to. It also reports `.tokengoatignore`, which applies to `token-goat pack` only: it excludes nothing from the symbol index. To keep a path out of the index, use `token-goat project exclude <path>`. |
 | `token-goat gdrive-sections <file-id>` | List the heading outline of a Google Doc without fetching the body. |
 | `token-goat stats` | See locally estimated savings: total events / bytes saved / tokens saved. Add `--full` for the per-source, per-command, and per-day breakdown, or `--methodology` to explain estimates and their limits. These values are not GitHub Copilot usage or billing data. |
 | `token-goat cost [--session]` | Estimated tokens saved, session or all-time, broken down by savings source. |
@@ -582,7 +587,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat skill-diff "<name>"` | Unified diff between the two most recent cached versions of a skill — tracks skill updates across sessions. |
 | `token-goat compact-hint --session-id <id>` | Inspect the compaction manifest for a session. Add `--trigger auto` to preview the pressure-aware budget the live PreCompact hook would use. |
 | `token-goat resume <session_id>` | Emit a single post-compact recovery packet — top skills, last two Bash outputs, top edited-file diffs, and `git diff --stat`, capped at ~2000 tokens. Replaces 5-10 round-trips. |
-| `token-goat config list / get / set / validate` | Inspect or edit `config.toml` from the CLI. `validate` reports unknown keys with did-you-mean suggestions, plus any project-file or environment value that validation rejected or clamped. A project-root `.token-goat.toml` layers on top of the global config, overriding hint thresholds, indexing settings, etc. for that project only; `config get`/`list`/`set` report which layer a value actually resolved from — the project file, an environment variable, or the global config — and where that layer's value was clamped or rejected they say so, naming what was asked for and what is in effect instead. |
+| `token-goat config list / get / set / validate` | Inspect or edit `config.toml` from the CLI. `validate` reports unknown keys with did-you-mean suggestions, plus any project-file or environment value that validation rejected or clamped. A project-root `.token-goat.toml` layers on top of the global config, overriding hint thresholds, indexing settings, etc. for that project only. It may not set the security sections `injection`, `webfetch`, `gdrive`, or `mcp`, nor `indexing.cross_project_symbols`: that file arrives with the repository, so a cloned project could otherwise switch off prompt-injection fencing or empty the fetch allow list for anyone who opened it. Those settings come from the global config or the environment only, and a project file that tries to set one is ignored with a message naming what was dropped. `config get`/`list`/`set` report which layer a value actually resolved from — the project file, an environment variable, or the global config — and where that layer's value was clamped or rejected they say so, naming what was asked for and what is in effect instead. |
 | `token-goat config-get <file> <key>` | Look up one key from a config-shaped file (TOML/INI `key = value`, or YAML) without reading the whole thing. On a `.md` file, a leading `---`-fenced YAML frontmatter block (Jekyll/Hugo/SKILL.md style) is checked first and takes precedence over the TOML/INI fallback; a `.md` file with no frontmatter, or an unclosed fence, falls through to the normal lookup unchanged. |
 | `token-goat pdf-extract <file>` | Extract plain text from a PDF instead of a raw Read. `--pages <spec>` narrows to a page range (e.g. `1-5` or `3`); `--head`/`--tail`/`--grep`/`--max-matches`/`--section` slice the extracted text the same way `bash-output`/`web-output` do. `--layout` heuristically reconstructs column-aware reading order from text-item coordinates instead of raw content-stream order (imperfect on rotated/overlapping text). |
 | `token-goat pdf-outline <file>` | List a PDF's bookmark/outline tree with page numbers instead of a raw Read. |
@@ -632,10 +637,10 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat hot [--limit N]` | Cross-session file frequency table: read and edit counts tallied from all stored sessions, ranked by total activity. Shows which files dominate your token spend across your entire history. `--project <dir>` filters to one project; `--json` for structured output. |
 | `token-goat note set/get/unset/list/clear` | Persistent per-project notes stored as key-value pairs. Token-goat injects them at session start and after compaction so they survive conversation rollover. Use to pin decisions, constraints, or reminders that would otherwise vanish after compaction. `note list --json` for machine-readable output; `note clear` removes everything at once. |
 | `token-goat project list` | Show all project roots indexed by token-goat with their file counts. Roots on the blocklist appear tagged `[excluded]`. `--json` for structured output. |
-| `token-goat project exclude <path>` | Add a project root to the blocklist so the worker never indexes it. Writes the resolved absolute path to `[worker] blocked_roots` in `config.toml`; idempotent. Remove the entry from the config to re-enable indexing. |
+| `token-goat project exclude <path>` | Add a project root to the blocklist so the worker never indexes it. Writes the resolved absolute path to `[worker] blocked_roots` in `config.toml`; idempotent. It also removes anything already indexed under that path and says how many files went, so excluding a directory means its contents stop being readable through `symbol` rather than merely stopping future indexing. Remove the entry from the config to re-enable indexing, then run `token-goat index` to bring the contents back. |
 | `token-goat project prune [--dry-run]` | Remove blocked/excluded roots that no longer exist on disk. `--dry-run` previews removals without touching the config file. Useful after deleting or moving projects. |
 | `token-goat install` | Wire up hooks (and, with the harness flags below, other AI tool integrations). No `--dry-run` or `--verify` flag — run `token-goat doctor` after install to audit the result. |
-| `token-goat doctor` | Confirm everything is wired correctly. Surfaces install state, cold-import timing, cache hit rates, compaction-budget telemetry, opt-in flag status, and canonical-root sanity. It read-only audits `~/.copilot/mcp-config.json` for globally configured Chrome DevTools or Playwright `npx` launchers, recommending project scope or removal when inactive; it never prints server configuration or secrets. On Windows, it also reports duplicate Chrome DevTools/Playwright MCP launchers and orphaned Node processes without terminating anything. Pass `--context` to show the **Context footprint** section: a fill bar with severity (ok / warn / high / URGENT), per-component breakdown (skills catalog, loaded skill bodies, CLAUDE.md+MEMORY.md, conversation estimate), session-to-session growth trend with sessions-to-URGENT projection, and tiered compaction recommendations (Tier 0–4) naming the exact commands to run. Auto-shown when fill > 40 % or any loaded skill > 2 K tokens lacks a compact. `--json` emits the check results (one entry per check, with `ok`/`warn`/`fail` status) as JSON instead of text. |
+| `token-goat doctor` | Confirm everything is wired correctly. Surfaces install state, cold-import timing, cache hit rates, compaction-budget telemetry, opt-in flag status, and canonical-root sanity. A **Security** section reports the posture in one place: whether offline mode is on, whether injection scanning is on, whether the Google Drive integration is enabled, whether fetching runs against an allow list or a deny list, whether MCP reads are confined to the project root, and whether the data directory is readable by other local users. It only warns when a protection that ships on has been switched off, so a default install stays quiet. It read-only audits `~/.copilot/mcp-config.json` for globally configured Chrome DevTools or Playwright `npx` launchers, recommending project scope or removal when inactive; it never prints server configuration or secrets. On Windows, it also reports duplicate Chrome DevTools/Playwright MCP launchers and orphaned Node processes without terminating anything. Pass `--context` to show the **Context footprint** section: a fill bar with severity (ok / warn / high / URGENT), per-component breakdown (skills catalog, loaded skill bodies, CLAUDE.md+MEMORY.md, conversation estimate), session-to-session growth trend with sessions-to-URGENT projection, and tiered compaction recommendations (Tier 0–4) naming the exact commands to run. Auto-shown when fill > 40 % or any loaded skill > 2 K tokens lacks a compact. `--json` emits the check results (one entry per check, with `ok`/`warn`/`fail` status) as JSON instead of text. |
 | `token-goat baseline` | Emit a project map: file count, per-language file counts, the top indexed symbols (by name/kind/location), and the most recently modified files. `--subagent` emits a terser variant (fewer symbols, fewer recent files) for context handed to a freshly spawned subagent; `--json` for the machine-readable form. |
 | `token-goat compact-doc <path>` | Build an extractive compact sidecar for a large reference doc (`.md`/`.markdown`). The compact is stored in the token-goat data dir as a SHA-keyed sidecar; `pre_read` serves it in place of the full file when it exists and is fresh, saving 80–95% of context tokens. Use `--force` to rebuild, `--sentences N` to control lines per section (default 2), `--show` to print the result. The sidecar is automatically marked stale when you edit the source file. Config: `[hints] stable_doc_compacts = true` (default on). |
 
@@ -883,13 +888,13 @@ never drift on where `mcp.json` lives or what key name it looks for.
 
 ## What gets installed?
 
-`token-goat install` writes the following on your machine — nothing else, anywhere. Every entry is reversed by `token-goat uninstall`. Run `token-goat doctor` at any time to see which of these are currently present.
+`token-goat install` writes the following on your machine — nothing else, anywhere. Every entry is reversed by `token-goat uninstall`. Integrations for other harnesses are additive on the way out as well as in, so a plain uninstall does not touch one you installed with `--codex`, `--copilot`, or a sibling flag: rather than undo something you did not ask about, it names each one still present and the flag that removes it. Run `token-goat doctor` at any time to see which of these are currently present.
 
 **Claude Code integration** (`~/.claude/`)
 
 | Path | What |
 |------|------|
-| `~/.claude/settings.json` | Hook entries for `SessionStart`, `PreToolUse` (Read/Grep/Bash, Drive/WebFetch), `PostToolUse` (Edit/Write/MultiEdit, Read/Grep/Glob, Bash, WebFetch, Skill), and `PreCompact`. Plus a `Bash(token-goat:*)` permission allowlist entry. Existing hooks are preserved; a timestamped `.bak` is written before any change.<br><br>The `PreToolUse` and `PostToolUse` matchers are narrowed to exactly the tools token-goat handles (plus `^mcp__`), generated from the live hook registry rather than a fixed list, so they can't fall out of date as handlers change. Claude Code starts a new process per matcher hit and most of that cost is process startup, so a catch-all matcher would make every unrelated tool call — `TodoWrite`, `TaskUpdate`, and friends — pay for a hook that has nothing to do. |
+| `~/.claude/settings.json` | Hook entries for `SessionStart`, `PreToolUse` (Read/Grep/Bash, Drive/WebFetch), `PostToolUse` (Edit/Write/MultiEdit, Read/Grep/Glob, Bash, WebFetch, Skill), and `PreCompact`. Hook entries only: install writes nothing under `permissions`, so it never grants the agent unprompted execution of anything. Existing hooks are preserved; a timestamped `.bak` is written before any change.<br><br>The `PreToolUse` and `PostToolUse` matchers are narrowed to exactly the tools token-goat handles (plus `^mcp__`), generated from the live hook registry rather than a fixed list, so they can't fall out of date as handlers change. Claude Code starts a new process per matcher hit and most of that cost is process startup, so a catch-all matcher would make every unrelated tool call — `TodoWrite`, `TaskUpdate`, and friends — pay for a hook that has nothing to do. |
 | `~/.claude/hooks/token-goat-shim.js` | The hook script those `settings.json` commands invoke (`"<node>" "<shim>" <event> "<entry>"`). It imports the hook library in-process instead of spawning a second process, and naming the node binary directly skips the npm bin wrapper — on Windows a `cmd.exe` layer every hook would otherwise pay for. Measured 480 ms → 324 ms per hook call. Regenerated on every `install` run. Always written here even for a `--project` install, since the command bakes in machine-specific absolute paths; a project-scope `settings.json` just points at this one. |
 | `~/.claude/CLAUDE.md` | A delimited block (`<!-- token-goat-begin -->` … `<!-- token-goat-end -->`) telling the agent to prefer `token-goat read` / `symbol` / `section` over `Read` / `Grep`. Any existing content is preserved. |
 | `~/.claude/skills/token-goat/SKILL.md` | The token-goat skill — the same routing guidance in skill form. |
@@ -908,6 +913,10 @@ There is no auto-update mechanism. Updating token-goat is always a manual `npm i
 
 Contains the symbol index (`global.db`, per-project `.db` files), session cache, shrunken-image cache, cached skill bodies (5 MB cap, LRU-evicted), logs, locks, and the dirty-file queue. Nothing outside this directory and `~/.claude/` is written.
 
+**What the index actually holds, in plain terms.** The point of a surgical read is returning a function body without the file around it, which means the database stores those bodies. `symbols.body` holds the source text of every indexed symbol, `symbols.docstring` its doc comment, `refs.context` the line around each reference, and `chunks.text` the passages that semantic search embeds. There is also a full-text index over the bodies and docstrings. So the database is not a list of names and line numbers: it is a substantial copy of your source, sitting in a plain unencrypted SQLite file outside the repository, at the path in the table above.
+
+Three things follow, and they are worth knowing before you decide. It never leaves the machine: token-goat sends no telemetry of any kind, and the only outbound requests it makes at all are the ones listed in the security section, none of which carry index content. It is not protected by your repository's access controls any more, so anything on the machine that can read your home directory can read it, and on Linux and macOS that directory sits under a home that backup and sync tools routinely copy. And it outlives an uninstall unless you say otherwise: `token-goat uninstall --purge` deletes both roots and tells you how much it reclaimed.
+
 **With `--codex`** (Codex CLI integration)
 
 | Path | What |
@@ -1194,20 +1203,41 @@ Outbound network is reserved to these explicit cases:
 - Image fetches from URLs: either explicit via `token-goat fetch-image <url>`, or when the AI agent issues a WebFetch call that returns image content — the hook intercepts and shrinks the image. The URL always originates from the agent's work, not from token-goat itself.
 - `token-goat screenshot <url>` navigates a headless browser to the URL you give it, subject to the target restrictions described below.
 - The first `token-goat semantic` run on a machine downloads the embedding model from `huggingface.co`, pinned to an immutable commit rather than a mutable branch. Subsequent runs use the local cache and make no network call. Skip the download entirely by setting `indexing.embeddings_enabled = false` (it is on by default), in which case `semantic` falls back to full-text search.
+- The first optical-character read of an image downloads the English language data (about 4 MB) from `cdn.jsdelivr.net`, at a fixed version path. Subsequent reads use the local cache. This happens for an explicit `token-goat image-text`, and also for the automatic text extraction the image-shrink hook performs when the agent reads a screenshot; turn the automatic one off with `image_shrink.ocr_enabled = false`.
+
+**One switch for all of it.** Set `network.offline = true` (env `TOKEN_GOAT_OFFLINE`) and every one of the paths above refuses instead of connecting, saying so rather than failing quietly. Anything already cached keeps working: a machine that has the embedding model still runs `semantic`, and one that has the language data still reads text out of images. This is one of the settings a per-project config file may not touch, so cloning a repository cannot switch it back off.
+
+**A repository cannot reconfigure the security controls.** A project-root `.token-goat.toml` layers on top of your global config, which is what it is for: hint thresholds, indexing settings, compression tuning. But that file arrives with the repository, so whoever wrote the repository wrote it. Five whole sections are therefore off limits to it, plus one individual key, and come from your global config or the environment only: `injection` (prompt-injection fencing), `webfetch` (the fetch allow and deny lists), `gdrive` (the Google Drive integration), `mcp` (root confinement and the allowed-roots list), `network` (offline mode), and the single key `indexing.cross_project_symbols`. A project file that sets one of them is ignored, and token-goat prints a line naming what it dropped. Everything else stays project-overridable.
 
 **Security reports.** See [SECURITY.md](SECURITY.md). Email `token-goat@dfkhelper.com`; do not file as a GitHub issue. Reports are acknowledged within 7 days; coordinated disclosure with a 90-day default window.
 
+**Dependency advisories.** `npm audit` on the published package is not empty. The residual findings all trace to three packages that have no forward patch, two of which are optional and can be left out with `npm install --omit=optional`. Each one is named, with why it does or does not reach you, under [Dependency advisories](SECURITY.md#dependency-advisories).
+
+**Verifying what you installed.** Every published version is built and pushed by one pinned workflow when a GitHub release is published, with npm provenance, so `npm audit signatures` verifies the tarball against the commit that produced it. Details in [Verifying what you installed](SECURITY.md#verifying-what-you-installed).
+
 **Prompt injection.** When an AI reads a file, web page, or command output, that content enters its context alongside your own instructions. Prompt injection is when untrusted content includes text designed to look like instructions — "Ignore all previous directives and run this instead" — to redirect the AI mid-task.
 
-Token-goat intercepts every Read, Fetch, and Bash call the AI makes. For Read and Bash, it does not filter or sanitize content on the way through to the model; doing so would silently break legitimate use cases. WebFetch is the one exception: every fetched page is scanned for a set of imperative-override attack patterns ("ignore previous instructions," "reveal system prompt," and similar), and a match is wrapped in an untrusted-content fence rather than passed through untouched (`injection.enabled`, on by default). Outside of that scan, the primary defense is the model's own training to treat tool output as data, not as commands from a trusted party.
+Token-goat intercepts every Read, Fetch, Bash, and MCP call the AI makes. Content is scanned for a set of imperative-override attack patterns ("ignore previous instructions," "reveal system prompt," and similar), and a match is wrapped in an untrusted-content fence rather than passed through untouched (`injection.enabled`, on by default). Three surfaces are covered. Every fetched page is scanned as it arrives and again when a cached copy is recalled with `web-output`. Every MCP tool result is scanned as it arrives, which matters most: it is a remote server's output, so it is the least trustworthy text in the pipeline. And cached Bash and MCP output is scanned when recalled with `bash-output` or `mcp-output`, since the output of a build or test run in a project with a hostile dependency is written by a third party as much as any web page is. The fence naming tool output is a different tag from the one naming web content, so the label tells the model where the text came from.
+
+Read is the exception: file content passes through to the model unfiltered, because filtering it would silently break legitimate use cases. Where token-goat splices a piece of a file into its own hint or denial message, that excerpt is fenced unconditionally. A fence only appears on a positive match, so ordinary output is unchanged. Outside of the scan, the primary defense is the model's own training to treat tool output as data, not as commands from a trusted party.
 
 Separately from that pass-through case: when a read hook *denies* a Read and substitutes its own message, any file bytes it embeds in that message (a markdown heading tree, a served compact or notebook sidecar, a re-read diff, a CSV header row, an HTML title) are wrapped in an `<untrusted-file-content>` fence first, so a hostile repo cannot get its own text presented to the model as token-goat speaking. That fencing is unconditional, not gated on the pattern scan.
 
-The MCP tools (`symbol` when given a `file` filter, `read`, `section`, `skeleton`, `outline`, `refs`, `brief`, `grep`, `imports`, `exports`) are confined to the project root, resolving symlinks before the check. Set `mcp.confine_reads_to_project_root = false` (env `TOKEN_GOAT_MCP_CONFINE_READS`) if you genuinely need cross-root reads from an MCP client. The CLI is deliberately unconfined and unchanged. This is defense in depth for one sink, not a sandbox: an agent that can call these tools can usually call its own read tool too.
+The MCP tools (`symbol` when given a `file` filter, `read`, `section`, `skeleton`, `outline`, `refs`, `brief`, `grep`, `imports`, `exports`) are confined to the project root, resolving symlinks before the check. Set `mcp.confine_reads_to_project_root = false` (env `TOKEN_GOAT_MCP_CONFINE_READS`) in your global config if you genuinely need cross-root reads from an MCP client; a per-project file cannot set it. The CLI is deliberately unconfined and unchanged. This is defense in depth for one sink, not a sandbox: an agent that can call these tools can usually call its own read tool too.
 
 Note what that flag does and does not cover. It stops a caller traversing *out of* the root it is given; it does not constrain *which* root the caller supplies. Every MCP tool takes an optional `projectRoot`, and it exists for a reason — the server's cwd is often not the workspace root for MCP clients — but tool arguments are model-generated, so that choice is untrusted input like any other. If your deployment treats MCP as the only path to the filesystem, set `mcp.allowed_roots` (env `TOKEN_GOAT_MCP_ALLOWED_ROOTS`, delimiter-separated like `PATH`) to the roots that may legitimately be named; a resolved root outside every entry is then refused. It is empty by default, which keeps the multi-root behaviour above unchanged.
 
-**Secret redaction in cached content.** Token-goat caches command output, fetched pages, and MCP results so it can serve them back later instead of re-running the work. Anything it writes to those caches is passed through a redactor first, so a credential that appeared in output does not sit on disk in plain text and does not get replayed into a later session. This is unconditional — there is no flag to turn it on, and it applies to cached Bash and Task output (including the command string itself, which is where an inline `--token=...` would otherwise land), fetched web content, MCP tool results and their labels, `compress-text`/`handoff` payloads, and the raw JSON disk cache. Recognized shapes: Anthropic, OpenAI, AWS, GitHub, Slack, Stripe, npm, and Google keys; JWTs; `Authorization: Bearer`/`Basic` headers; PEM private-key blocks; and generic `password=`/`secret=`/`api_key=` assignments in `.env`, connection-string, and query-string shape. A match is replaced by a `[REDACTED:<kind>]` marker naming which pattern fired.
+**Restricting what token-goat may fetch.** `webfetch.allow` and `webfetch.deny` (env `TOKEN_GOAT_WEBFETCH_ALLOW` / `TOKEN_GOAT_WEBFETCH_DENY`, comma-separated) are wildcard URL patterns that decide which addresses may be reached. Deny is checked first and wins; a non-empty allow list refuses anything it does not name. Patterns are matched against the address as it will actually be sent, not only as you typed it, so a trailing dot on the host, `..` path segments, a default port written out, and percent-encoded path characters cannot be used to step around a rule. Writing a default port in a pattern (`https://example.com:443/*`) and omitting it are equivalent. Both are empty by default, which permits everything, exactly as before. They apply to the WebFetch call your AI makes, to the fetches token-goat performs itself (`fetch-image`, `gdrive-sections`), and to the headless browser behind `screenshot` (whose page sub-resources are checked too), including every redirect hop, so an allowed site cannot redirect the request on to a denied one.
+
+**Where cached content lives, and who can read it.** Cached command output, fetched pages, MCP results, session state and the source index all sit under one data directory (`~/.local/share/token-goat` on Linux, `~/Library/Application Support/token-goat` on macOS, `%LOCALAPPDATA%\dfk-helper\token-goat` on Windows). On POSIX that directory is created owner-only (mode 0700), and an existing one is tightened on the next run, so other local users on a shared build host cannot read it. Windows uses inherited ACLs instead. Individual JSON blobs are additionally written 0600.
+
+**Confining `symbol` to one project.** `token-goat symbol` is the one read command that answers from the machine-wide index (`global.db`) rather than the current project, so by default `symbol <name>` and `symbol --grep .` return matching symbols, bodies included, from every project ever indexed on the host. That is deliberate and useful on a personal machine: it is how you find a helper you wrote in another repo. On a shared build host, or under an agent you have confined to one directory, it is a read channel that the directory sandbox does not close, because the answer comes out of the index instead of the filesystem. Set `indexing.cross_project_symbols = false` (env `TOKEN_GOAT_CROSS_PROJECT_SYMBOLS`) and `symbol` only answers from the project it is run in. `--project` and `--file` pointing outside that project are refused rather than honored, so the setting cannot be stepped around from inside the confined process. Every other read command (`read`, `refs`, `callers`, `types`, `dead`, `find`, `semantic`) is already project-scoped and is unaffected.
+
+**Turning off the Google Drive integration.** `token-goat gdrive-sections` is the only feature that talks to Google. Set `gdrive.enabled = false` (env `TOKEN_GOAT_GDRIVE_ENABLED`) and the command refuses before it opens a connection, and the routing guidance token-goat writes into CLAUDE.md, AGENTS.md, `copilot-instructions.md` and the installed skill stops naming it, so an agent is never told the command exists. Nothing else in token-goat contacts Google Drive, and it holds no Drive credentials: `gdrive-sections` fetches the public export URL of a document id you pass it by hand.
+
+**Secret redaction in cached content.** Token-goat caches command output, fetched pages, and MCP results so it can serve them back later instead of re-running the work. Anything it writes to those caches is passed through a redactor first, so a credential that appeared in output does not sit on disk in plain text and does not get replayed into a later session. This is unconditional — there is no flag to turn it on, and it applies to cached Bash and Task output (including the command string itself, which is where an inline `--token=...` would otherwise land), fetched web content, MCP tool results and their labels, `compress-text`/`handoff` payloads, and the raw JSON disk cache. Recognized shapes: Anthropic, OpenAI, AWS, GitHub, Slack, Stripe, npm, and Google keys; JWTs; `Authorization: Bearer`/`Basic` headers; PEM private-key blocks; presigned-url signatures (AWS `X-Amz-Signature`, Google Cloud Storage `X-Goog-Signature`, Azure SAS `sig`); and generic `password=`/`secret=`/`api_key=` assignments in `.env`, connection-string, and query-string shape. A match is replaced by a `[REDACTED:<kind>]` marker naming which pattern fired.
+
+Session state gets the same treatment. The per-session file records which urls were fetched and which `curl -o` downloads landed where, and a url carries credentials as readily as output does. The fetched-url list is redacted, so the compaction manifest can still name what was fetched without naming the key; the download list is keyed by a digest of the url instead, because its only consumer is an exact-match check and a redaction there would make two urls differing only in their key look identical. The fetched-url entry also redacts the prompt that was sent with the page, and carries a digest of the pair so redaction cannot merge two entries that differ only inside the redacted span. An entry written by an older version is rewritten into this shape the first time the file is read, so upgrading clears the credentials an old file was holding rather than keeping them for the life of the session.
 
 Two honest limits. It is a pattern matcher, not a classifier: a credential in a format it does not recognize — an internal token shape, a bare high-entropy string with no `key=` prefix — is cached as-is. And it protects what token-goat *stores*, not what your agent reads in real time; a secret printed to the terminal was already in the model's context before any caching happened. Treat it as damage control on the cache layer, not a reason to relax about printing secrets.
 
@@ -1229,13 +1259,22 @@ Add-MpPreference -ExclusionPath "$env:LOCALAPPDATA\dfk-helper\token-goat"
 token-goat uninstall
 ```
 
-Reverses everything in [What gets installed?](#what-gets-installed): the hook entries in `settings.json`, the `CLAUDE.md` block, the skill directory. Add `--codex`, `--gemini`, `--opencode`, `--pi`, `--hermes`, `--openclaw`, `--copilot`, `--grok`, or `--vscode` to also strip those integrations. There is no `--purge` flag — `uninstall` never deletes the data directory (cache, index, models, logs); remove it by hand if you want it gone. It also does not stop a running worker; use `token-goat worker stop` for that. Nothing else on the system depends on it.
+Reverses everything in [What gets installed?](#what-gets-installed): the hook entries in `settings.json`, the `CLAUDE.md` block, the skill directory. Add `--codex`, `--gemini`, `--opencode`, `--pi`, `--hermes`, `--openclaw`, `--copilot`, `--grok`, or `--vscode` to also strip those integrations. It does not stop a running worker; use `token-goat worker stop` for that. Nothing else on the system depends on it.
+
+By default the data directories stay: the index took real time to build and a reinstall wants it back. Add `--purge` to delete them as well, which is what offboarding a machine needs:
+
+```
+token-goat worker stop
+token-goat uninstall --purge
+```
+
+That removes both roots (the data directory holding the index, caches, models and logs, and the home directory holding session state and the OCR cache), naming each one and how much it reclaimed. It refuses while the worker is running, because the worker would rewrite files under a directory being deleted.
 
 ## About
 
 I built this because long Claude Code and Codex sessions on my machine kept burning context in the same ways: screenshots landing at 2-3 MB, the agent re-reading a file it parsed hours earlier in the same conversation, compactions that forgot which functions were edited. Each felt preventable.
 
-This is a solo project. I use it daily on Windows 11. Tests run on Node.js 20 and 22.
+This is a solo project. I use it daily on Windows 11. The full test suite runs on Node.js 22 across Windows, macOS, and Linux, and all three must pass before anything merges.
 
 ## Sister project
 

package/SECURITY.md

@@ -0,0 +1,136 @@
+# Security Policy
+
+Token-Goat runs on your machine, registers hooks with two AI coding tools, and reads files those tools open. The attack surface is real and the project treats security reports as a priority.
+
+## Reporting a vulnerability
+
+Email token-goat@dfkhelper.com. This is a private inbox, not a public issue tracker. Do not file security reports as GitHub issues; that exposes the finding before a fix ships. PGP key available on request.
+
+A useful report contains:
+
+- Affected Token-Goat version (`token-goat --version`)
+- Operating system and Node version (`node --version`)
+- Reproduction steps, ideally a minimal command sequence
+- Observed impact and a short severity assessment
+- Suggested fix, if known
+
+## What to expect
+
+Reports are acknowledged within 7 calendar days of receipt. If you have not heard back in that window, resend; mail does get lost. After triage, a target fix window is set based on severity and communicated back. Coordinated disclosure is preferred, with a typical 90-day window before public details. Reporters who want public credit are credited in the changelog and the release notes. Reporters who prefer to stay anonymous are kept anonymous.
+
+## In scope
+
+The following are treated as security issues:
+
+- Privilege escalation through Token-Goat's installer, worker, or hooks
+- Remote code execution via hook payloads, CLI arguments, or cached content
+- Data exfiltration through Token-Goat's database, cache, or session store
+- Injection vulnerabilities in any user-facing command or hook input path
+- Supply-chain concerns affecting the published `token-goat` package
+- Authentication or authorization flaws in token-bearing integrations
+
+## Out of scope
+
+The following are not treated as security issues unless paired with a working proof of concept showing actual impact:
+
+- Theoretical vulnerabilities without a reproducer
+- Issues in upstream dependencies that do not manifest through Token-Goat's surface
+- Local denial of service via resource exhaustion (memory, disk, CPU) on the user's own machine
+- Social-engineering attacks that require tricking the user into running malicious commands
+- Issues that require an already-compromised local user account
+
+## Dependency advisories
+
+`npm audit` gives three different answers for Token-Goat depending on what you scan, and the gap between them is the first thing to understand. All three numbers below are reproducible with the commands shown.
+
+| What you scan | Command | Result |
+| --- | --- | --- |
+| this repository | `npm audit` | clean, development dependencies included |
+| an install without optional packages | `npm install --omit=optional token-goat` then `npm audit --omit=dev --omit=optional` | clean, 46 packages |
+| a default install | `npm install token-goat` then `npm audit --omit=dev` | the optional packages in the table below |
+
+The repository is clean because [`package.json`](package.json) carries an `overrides` block that pins five transitive packages to patched versions. **npm applies `overrides` only in the root project**, so those pins do not travel to anyone who installs Token-Goat as a dependency. We are saying so plainly rather than letting a clean repository scan stand in for a clean install: if your scanner reads this repository or its lockfile it will report nothing, and that is not the whole picture.
+
+Without the overrides a consumer resolves `protobufjs` at 6.x, which carries more advisories than the 7.x line the repository would otherwise use. Every one of those paths is optional.
+
+| Package | Advisories it carries | Where it loads | Why the advisory does not reach you through Token-Goat |
+| --- | --- | --- | --- |
+| `@xenova/transformers` | [`protobufjs`](https://github.com/advisories/GHSA-xq3m-2v4x-88gg) (critical), `onnx-proto`, `onnxruntime-web`, and its own pinned `sharp` | optional; loaded only when semantic search builds or queries embeddings | not mitigated, so it is the one to weigh. Skip it with `npm install --omit=optional`, or leave `indexing.embeddings_enabled` off, and the code never loads |
+| `exceljs` | [`uuid`](https://github.com/advisories/GHSA-w5hq-g745-h8pq) | optional; loaded only when an `xlsx-*` command opens a workbook | the advisory is a missing bounds check on a caller-supplied `buf` argument; ExcelJS never passes one |
+
+`npm install --omit=optional` gives you an install without either of these, and `npm audit` reports it clean. Every command still starts; the ones that need a package you skipped say so. The `xlsx-*` commands report that ExcelJS is not installed rather than failing oddly, `zip-list`/`zip-read` do the same for fflate, and `semantic` keeps working on keyword search alone: it is the embedding half that goes away, not the command.
+
+`html-to-text` used to appear in that table, carrying [`deepmerge-ts`](https://github.com/advisories/GHSA-ggr8-5vv4-36mx). It is now a development dependency instead. esbuild inlines it into `dist/token-goat.mjs` at build time and nothing in the published bundle imports it, so it was a runtime dependency in name only: moving it removes `html-to-text`, `deepmerge-ts`, `htmlparser2`, `selderee`, and `dom-serializer` from an installed copy while the HTML-to-text output stays byte-for-byte identical. That is what takes the no-optional install to zero, and it is better than the alternative we had considered, rolling `html-to-text` back to 9.x: that version pins `htmlparser2` two majors lower, and `htmlparser2` is what parses fetched pages, so it would have traded an advisory in an options merger for an older parser on the one path that handles untrusted input.
+
+Direct dependencies with a forward patch are kept current rather than pinned: `sharp` and `puppeteer-core` were both moved across a major version to clear their advisories.
+
+For a scanner that ingests a bill of materials rather than a lockfile, `npm run sbom` writes CycloneDX 1.5 to stdout.
+
+## Verifying what you installed
+
+Every published version is built and pushed by one workflow, [`.github/workflows/publish.yml`](.github/workflows/publish.yml), which runs only when a GitHub release is published (or manually, and then only from `main`). It publishes with npm provenance, so npm holds a signed attestation tying the tarball to the commit and workflow run that produced it. Nothing is ever published from a laptop.
+
+To check a copy you already have:
+
+```
+npm audit signatures
+```
+
+Run from a project that depends on token-goat, that command verifies the registry signature and the provenance attestation for every installed package, token-goat included. The package page on npm links the attestation to the exact commit, so you can read the source that produced the bytes you are running.
+
+Every action used by that workflow, and by CI, is pinned to a full commit SHA rather than a tag, so a compromised action repository cannot silently change what runs. That is enforced by a test rather than by review: see [tests/guards/workflow_actions_pinned.test.ts](tests/guards/workflow_actions_pinned.test.ts).
+
+## License
+
+Token-Goat is source-available under the PolyForm Noncommercial License 1.0.0. Submitting a security report does not grant the reporter any license to Token-Goat's code beyond what PolyForm Noncommercial already permits. See LICENSE for the full terms.
+
+
+## Dependency licenses
+
+Every production dependency is permissively licensed, but a scan does not read it that way on its
+own. Counted from `package-lock.json`, which lists the packages for every platform rather than only
+the ones this machine installed, 22 entries need a human answer: 7 declare a license a scanner
+cannot resolve, and 15 carry a copyleft term. All 22 arrive through optional dependencies. Install
+with `npm install --omit=optional token-goat` and not one of them is present.
+
+**Declarations a scanner cannot resolve.** Both are upstream mistakes, and both are the same
+mistake this project made in its own manifest and fixed. `SEE LICENSE IN <file>` is npm's form for
+a license that is not on the SPDX list, and `MIT OR Apache` is not a valid expression because the
+identifier is `Apache-2.0`. For the second, `npm sbom` emits no `licenses` field at all rather than
+an unresolvable one.
+
+| Package | Declares | Actually grants | Reached through |
+| --- | --- | --- | --- |
+| `flatbuffers` | `SEE LICENSE IN LICENSE.txt` | Apache-2.0, in its own `LICENSE.txt` | `@xenova/transformers` |
+| `sqlite-vec` and its 5 platform packages | `MIT OR Apache` | MIT or Apache-2.0, your choice | direct optional dependency |
+
+**Copyleft terms.** Two families, and neither puts a copyleft obligation on Token-Goat's own code.
+
+| Package | Declares | Why it is not a problem |
+| --- | --- | --- |
+| `@img/sharp-libvips-<platform>` (10 packages) | `LGPL-3.0-or-later` | libvips, shipped as a prebuilt shared library and used unmodified. LGPL asks that the library stay replaceable, and it is: it is a separate package that `sharp` loads at runtime. |
+| `@img/sharp-<platform>` (4 packages) | `Apache-2.0 AND LGPL-3.0-or-later` | the Apache half is `sharp` itself, the LGPL half is the same libvips |
+| `jszip` | `(MIT OR GPL-3.0-or-later)` | a choice between the two, and the MIT half is taken |
+
+`sharp` and `jszip` are both optional: `sharp` powers image shrinking, `jszip` arrives through
+`exceljs` for the `xlsx-*` commands.
+
+**Three packages with no license at all used to be here.** `buffers@0.1.1` and `chainsaw@0.1.0`
+shipped with neither a `license` field nor a license file, and `traverse@0.3.9` had the file but
+not the field. No grant at all is worse for a review than a copyleft grant, because there is
+nothing to apply policy to. They arrived through `exceljs`, which depends on `unzipper`, which
+depended on `binary`, which depended on all three. `unzipper` dropped `binary` in 0.11, so an
+override to `^0.12.5` removes the sub-chain, and the deprecated `fstream` with it. Token-goat reads
+workbooks through `wb.xlsx.readFile()`, which goes through `jszip`; `unzipper` is only reachable
+through the streaming reader in `exceljs`, which this code never calls.
+
+Reproduce the whole picture:
+
+```bash
+npm run sbom
+```
+
+Like the advisory disclosure above, this is checked by a test rather than by review: see
+[tests/guards/dependency_licenses.test.ts](tests/guards/dependency_licenses.test.ts). Any
+production package that a scanner cannot resolve, or that carries a copyleft term, has to be named
+here or that test fails.

package/dist/{token-goat-hook-chunk-KNVHTGEX.mjs → token-goat-chunk-3KW5C2KQ.mjs}

@@ -2,7 +2,7 @@ import { createRequire as __cjsRequire } from 'node:module';
 const require = __cjsRequire(import.meta.url);
 import {
   init_define_import_meta_env
-} from "./token-goat-hook-chunk-7WC2H3EW.mjs";
+} from "./token-goat-chunk-3UOG7PSZ.mjs";
 
 // src/reset.ts
 init_define_import_meta_env();

package/dist/{token-goat-hook-chunk-7WC2H3EW.mjs → token-goat-chunk-3UOG7PSZ.mjs}

@@ -12,11 +12,20 @@ var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require
   if (typeof require !== "undefined") return require.apply(this, arguments);
   throw Error('Dynamic require of "' + x + '" is not supported');
 });
-var __esm = (fn, res) => function __init() {
-  return fn && (res = (0, fn[__getOwnPropNames(fn)[0]])(fn = 0)), res;
+var __esm = (fn, res, err) => function __init() {
+  if (err) throw err[0];
+  try {
+    return fn && (res = (0, fn[__getOwnPropNames(fn)[0]])(fn = 0)), res;
+  } catch (e) {
+    throw err = [e], e;
+  }
 };
 var __commonJS = (cb, mod) => function __require2() {
-  return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
+  try {
+    return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
+  } catch (e) {
+    throw mod = 0, e;
+  }
 };
 var __export = (target, all) => {
   for (var name in all)

package/dist/{token-goat-hook-chunk-FKZ5RBT4.mjs → token-goat-chunk-4M7PSAO3.mjs}

@@ -3,7 +3,7 @@ const require = __cjsRequire(import.meta.url);
 import {
   __export,
   init_define_import_meta_env
-} from "./token-goat-hook-chunk-7WC2H3EW.mjs";
+} from "./token-goat-chunk-3UOG7PSZ.mjs";
 
 // node_modules/zod/index.js
 init_define_import_meta_env();
@@ -590,7 +590,7 @@ function $constructor(name, initializer3, params) {
   Object.defineProperty(_, "name", { value: name });
   return _;
 }
-var $brand = Symbol("zod_brand");
+var $brand = /* @__PURE__ */ Symbol("zod_brand");
 var $ZodAsyncError = class extends Error {
   constructor() {
     super(`Encountered Promise during synchronous parse. Use .parseAsync() instead.`);
@@ -10407,8 +10407,8 @@ function yo_default() {
 // node_modules/zod/v4/core/registries.js
 init_define_import_meta_env();
 var _a2;
-var $output = Symbol("ZodOutput");
-var $input = Symbol("ZodInput");
+var $output = /* @__PURE__ */ Symbol("ZodOutput");
+var $input = /* @__PURE__ */ Symbol("ZodInput");
 var $ZodRegistry = class {
   constructor() {
     this._map = /* @__PURE__ */ new WeakMap();
@@ -11446,7 +11446,7 @@ function _stringbool(Classes, _params) {
     type: "pipe",
     in: stringSchema,
     out: booleanSchema,
-    transform: (input, payload) => {
+    transform: ((input, payload) => {
       let data = input;
       if (params.case !== "sensitive")
         data = data.toLowerCase();
@@ -11465,14 +11465,14 @@ function _stringbool(Classes, _params) {
         });
         return {};
       }
-    },
-    reverseTransform: (input, _payload) => {
+    }),
+    reverseTransform: ((input, _payload) => {
       if (input === true) {
         return truthyArray[0] || "true";
       } else {
         return falsyArray[0] || "false";
       }
-    },
+    }),
     error: params.error
   });
   return codec2;