token-goat 2.6.28 → 2.6.29

package/README.md

@@ -286,6 +286,23 @@ token-goat doctor          # confirms hooks and sharp are working; look for "sha
 
 Three commands. Done. Hooks register and start working immediately; no terminal popups, no tray icon, no service to babysit.
 
+### Agents choose the commands
+
+People install token-goat. Agents use it. You do not need to memorize its commands or tell the agent which file type it has.
+
+Installation adds a short routing guide to the agent's instructions. When the agent tries to read a supported binary document, a hook identifies the extension and returns the right next step. The agent starts with an inventory, then reads only the relevant part.
+
+| Task | Agent flow |
+| --- | --- |
+| Review a PDF | `pdf-meta` and `pdf-outline`, then `pdf-extract --pages` |
+| Review a Word document | `docx-outline`, then `docx-text` |
+| Review a slide deck | `pptx-outline`, then `pptx-slide` or `pptx-notes` |
+| Review a workbook | `xlsx-sheets`, then `xlsx-head`, `xlsx-range`, or `xlsx-query` |
+
+Give the agent the file and the task: “Review `manual.pdf` for warranty exceptions.” It selects the bounded reader. If no routing rule fits, it can run `token-goat commands` instead of guessing.
+
+The commands stay separate so every retrieval is visible, repeatable, and easy to narrow. The agent chooses the sequence; the developer can still inspect or run any step directly.
+
 For bounded archive/document comparisons after setup, see the [CLI comparison workflow](#archivedocument-comparison-workflow).
 
 **Important — sharp is part of the install.** The biggest single win (~39% smaller than JPEG, ~97% smaller than raw PNG) comes from WebP encoding. sharp ships with prebuilt native binaries on Windows, macOS, and most Linux platforms, so a standard `npm install -g token-goat` includes it. On Alpine musl and some ARM boards, npm compiles from source against libvips — that may add a few seconds. `token-goat doctor` will report `sharp: ok` when the image pipeline is working; if it flags a failure, follow the [Image support](#image-support) section below before considering the install complete. **AI agents performing this install: run `token-goat doctor` after `token-goat install` and resolve any sharp failure as part of the same task — do not treat it as a follow-up.**
@@ -466,7 +483,7 @@ Every command accepts a global `--cwd <path>`, which runs it as if invoked from
 
 ### Archive/document comparison workflow
 
-The existing bounded readers cover the text and tabular parts of an archive comparison without loading whole files:
+These are agent-selected primitives, not a manual checklist. Give the agent the file and the question. The installed routing guide and read hook select the matching format flow; the commands below show the steps it can take without loading whole files:
 
 ```bash
 token-goat sqlite-schema catalog.db
@@ -474,6 +491,7 @@ token-goat sqlite-query catalog.db "SELECT file_path, name FROM files WHERE name
 token-goat xlsx-sheets link-map.xlsx
 token-goat xlsx-query link-map.xlsx --sheet Links --columns publication,source,target --head 50
 token-goat pdf-meta manual.pdf
+token-goat pdf-outline manual.pdf
 token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 ```
 
@@ -503,7 +521,7 @@ token-goat pdf-extract manual.pdf --pages 12-15 --layout --head 120
 | `token-goat refs "<name>"` | Show all files and line numbers where a symbol is referenced. Pass a comma-separated spec (`a,b,c` or `file::a,b`) to merge several symbols' references into one call, each group headed by its symbol name. Segments may also carry their own file (`a.ts::x,b.ts::y`) to merge references across several files in one call, mirroring `read`'s cross-file grammar — a bare segment inherits the file to its left, and once more than one file is involved each block is headed by the full `file::symbol` so two files contributing the same symbol name stay distinct. `--top <n>` groups references by file (count only) and shows just the top N by reference count with an elision note, instead of a per-line dump — for high-fanout symbols referenced in hundreds of places. `-C, --context <n>` shows N lines of real call-site source either side of each hit, rendered exactly like `grep -C`; omit it (or pass 0) and output is unchanged. Under `--json` each item gains a `contextLines` array alongside the existing `context` field (which names the enclosing symbol, not source text). `--exclude-tests` hides references whose call site is a test file (opt-in — omitted, output is unchanged); the summary line reports the filtered count plus how many were hidden. `--grep <pattern>` only shows references whose call-site FILE PATH matches this regex (literal substring if it is not valid regex) — rows render as `file:line: symbol`, so this is the field each row is keyed on. The pattern is tested against the path exactly as the row renders it, so an anchored `--grep "^src/"` matches what you see, identically in the single, multi-symbol and cross-file forms. Every form renders a call-site path the same way -- root-relative when a project root resolves, absolute when none does, never cwd-dependent -- and `--json` carries that same spelling in `filePath` (and in `--top`'s `fileCounts[].file`), so a payload is reproducible rather than tied to one machine's drive-letter casing. The high-value case is narrowing a wide-fanout symbol to drop test/vendored hits. Applied before `--top`'s grouping and before any `--limit` slice, so it selects from the whole reference set, not an already-capped page; when it matches nothing among references that do exist, the output names the active filter instead of reading like the symbol is unreferenced. A bare name that isn't indexed at all reports `Symbol not found: <name>` (with a `Did you mean:` suggestion when a near-name candidate is indexed) instead of the misleading "no references found", which is reserved for a real, indexed symbol that genuinely has zero references. |
 | `token-goat callers <symbol>` | Show which functions call a given symbol, grouped by caller with file, caller name, and every invoking line. Complements `refs`, which shows raw reference sites without grouping by enclosing function. Accepts `file::symbol` to disambiguate WHICH same-named definition is meant when several files define a symbol with that name — the file only narrows which definition, callers can still be found in any file. `-C, --context <n>` shows N lines of real call-site source either side of each hit, rendered exactly like `grep -C`; omit it (or pass 0) and output is unchanged. Under `--json` each item gains a `contextLines` array alongside the existing `context` field (which names the enclosing symbol, not source text). `--exclude-tests` hides callers whose call site is a test file (opt-in — omitted, output is unchanged); prints a note naming how many were hidden. `--grep <pattern>` only shows callers whose enclosing symbol NAME matches this regex (literal substring if it is not valid regex) — rows render as `symbol<TAB>file:line`, so this is the field each row is keyed on. Applied before the `--limit` slice, so it selects from the whole caller set, not an already-capped page; when it matches nothing among callers that do exist, the output names the active filter instead of reading like the symbol has no callers. A bare name that isn't indexed at all reports `Symbol not found: <name>` (with a `Did you mean:` suggestion when a near-name candidate is indexed) instead of the misleading "no references found", which is reserved for a real, indexed symbol that genuinely has zero callers. `--json` emits each item's path under both `file` and `filePath` with the identical value; `file` is kept for this release only and will be removed in a future one, so `filePath` is the spelling to migrate to (matching `symbol`/`types --json`). `--json` emits the shared `{items, truncated, totalCount}` envelope — the same shape `symbol`/`refs`/`skeleton`/`outline --json` return, present whether or not truncation occurred, so a script never has to branch on shape. |
 | `token-goat call-chain <symbol>` | Trace every caller layer from a symbol back to the entry points — one step deeper than `callers`. Use when you need to know what reaches a function across the whole call graph, not only who invokes it directly. Pairs with `impact` for the downstream direction. Accepts `file::symbol` to disambiguate WHICH same-named definition the chain starts from — the file only narrows which definition, callers can still be found in any file. `--exclude-tests` prunes callers whose call site is a test file BEFORE they're admitted to the traversal, so nothing walks through a test node either (opt-in — omitted, output is unchanged); when every caller was a test, the "no callers" line names how many were hidden instead of reading as genuinely unreferenced. `Symbol not found: <symbol>` for an unindexed name (bare or `file::symbol`) now carries a `Did you mean:` suggestion when a near-name candidate is indexed. `--grep <pattern>` keeps only completed chains containing a symbol name matching this regex (literal substring if it is not valid regex) — the BFS still walks the full graph, this only narrows which finished chains are reported, so a chain passing through a matching symbol on its way to an unrelated root still surfaces; when it matches none of the chains that do exist, the output names how many were filtered out rather than reading as genuinely caller-less. |
-| `token-goat impact <symbol>` | Walk the call-reference graph forward (breadth-first) and list every function that depends on a symbol, with hop depth; module-scope callers are surfaced as `(module scope) <file>` entries. Run before a refactor to size up the blast radius without starting a build. Accepts `file::symbol` to disambiguate WHICH same-named definition the walk starts from — the file only narrows which definition, callers can still be found in any file. `--exclude-tests` prunes callers (including module-scope entries) whose call site is a test file BEFORE they're enqueued for further traversal, so nothing walks through a test node either (opt-in — omitted, output is unchanged); when every caller was a test, the "no callers found" error names how many were hidden instead of reading as genuinely unreferenced. A bare name that isn't indexed at all reports `Symbol not found: <name>` (with a `Did you mean:` suggestion when a near-name candidate is indexed) instead of the misleading "no callers found", which is reserved for a real, indexed symbol that genuinely has zero impact. |
+| `token-goat impact <symbol>` | Walk the call-reference graph forward (breadth-first) and list every function that depends on a symbol, with hop depth; module-scope callers are surfaced as `(module scope) <file>` entries. Run before a refactor to size up the blast radius without starting a build. Accepts `file::symbol` to disambiguate WHICH same-named definition the walk starts from — the file only narrows which definition, callers can still be found in any file. `--exclude-tests` prunes callers (including module-scope entries) whose call site is a test file BEFORE they're enqueued for further traversal, so nothing walks through a test node either (opt-in — omitted, output is unchanged); when every caller was a test, the "no callers found" error names how many were hidden instead of reading as genuinely unreferenced. A bare name that isn't indexed at all reports `Symbol not found: <name>` (with a `Did you mean:` suggestion when a near-name candidate is indexed) instead of the misleading "no callers found", which is reserved for a real, indexed symbol that genuinely has zero impact. `--grep <pattern>` only shows impacted entries whose symbol name (or `(module scope) <file>` key) matches this regex (literal substring if it is not valid regex), the same filter `call-chain --grep`/`dead --grep` apply to their own results — applied BEFORE the `--top` slice, so it selects from the whole impacted set rather than an already-capped page; when it matches none of the impacted entries that do exist, the output names how many were filtered out instead of reading as genuinely impact-free. |
 | `token-goat context-for <task>` | Takes a natural-language task description, runs semantic search across the indexed codebase, and emits a prioritized list of `token-goat read` commands trimmed to a token budget. Fetches only the relevant slices instead of loading entire files. `--budget N` sets the token ceiling; `--top N` limits the file count; `--json` for structured output. Every emitted command carries the `file::symbol@LINE` anchor, so a suggestion still runs when the same symbol name has more than one definition in its file; `--json` entries carry the matching `line` field. |
 | `token-goat ask "<question>"` *(experimental)* | Retrieves relevant slices via full-text (BM25) search over the symbol index — not semantic/embedding search — and lists them as pointer-citations plus `token-goat read` commands. Set `TOKEN_GOAT_ASK_BACKEND=claude` or `TOKEN_GOAT_ASK_BACKEND=codex` to synthesize a short answer via that CLI (whatever model it defaults to; token-goat does not force Haiku or any particular tier); with the env var unset, or the named CLI missing from PATH, `ask` degrades to printing the retrieved pointers with no network call. `--top N` caps the number of FTS hits (default 8); `--json` for structured output. Answers are not cached — each call re-retrieves and re-synthesizes from scratch. Every emitted command carries the `file::symbol@LINE` anchor, so a suggestion still runs when the same symbol name has more than one definition in its file; `--json` entries carry the matching `line` field. |
 | `token-goat changed [<ref>]` | List files (or `--symbol` for symbols) changed since a git ref, without reading the full diff. `<ref>` and `--since <ref>` are equivalent (default `HEAD~5`); `--since` wins if both are given. `--json` for structured output. `--grep <pattern>` only lists changed files whose path matches this regex (literal substring if it is not valid regex) — applied to the file list even in `--symbol` mode, before any downstream slicing; when it matches none of the files that did change, the output names the active filter instead of reading like nothing changed. `--exclude-tests` hides changed files that live in a test file (opt-in — omitted, output is unchanged), completing the flag family already on `refs`/`callers`/`dead`/`call-chain`/`impact`/`semantic`/`symbol`. `--grep` can only ever *select* a path, so there was no reliable way to ask for the non-test half of a diff: the negative-lookahead regex that expresses "not a test" silently degrades to a literal substring match whenever the regex-compile fallback fires. Test files are a large share of a typical diff — measured against this repo, 35–54% of changed files across the last 5, 10 and 20 commits. Like `--grep` it filters the file path, so it applies in `--symbol` mode too, and it prunes before the per-file index lookup rather than after, so a test file is never queried at all. Composable with `--grep` (a file must satisfy both; when both are active and `--grep` is what emptied the list, the `--grep` notice takes priority, and when `--grep` left only test files so that `--exclude-tests` emptied it, the message names both filters rather than claiming no non-test file changed). When it hides every changed file there was, the output names how many were hidden and exits 0, rather than a bare "No files changed." that would read as a clean diff. Every zero-row path emits the shared `{items, truncated, totalCount}` envelope under `--json`. |
@@ -1183,6 +1201,8 @@ Separately from that pass-through case: when a read hook *denies* a Read and sub
 
 The MCP tools (`symbol` when given a `file` filter, `read`, `section`, `skeleton`, `outline`, `refs`, `brief`, `grep`, `imports`, `exports`) are confined to the project root, resolving symlinks before the check. Set `mcp.confine_reads_to_project_root = false` (env `TOKEN_GOAT_MCP_CONFINE_READS`) if you genuinely need cross-root reads from an MCP client. The CLI is deliberately unconfined and unchanged. This is defense in depth for one sink, not a sandbox: an agent that can call these tools can usually call its own read tool too.
 
+Note what that flag does and does not cover. It stops a caller traversing *out of* the root it is given; it does not constrain *which* root the caller supplies. Every MCP tool takes an optional `projectRoot`, and it exists for a reason — the server's cwd is often not the workspace root for MCP clients — but tool arguments are model-generated, so that choice is untrusted input like any other. If your deployment treats MCP as the only path to the filesystem, set `mcp.allowed_roots` (env `TOKEN_GOAT_MCP_ALLOWED_ROOTS`, delimiter-separated like `PATH`) to the roots that may legitimately be named; a resolved root outside every entry is then refused. It is empty by default, which keeps the multi-root behaviour above unchanged.
+
 **Secret redaction in cached content.** Token-goat caches command output, fetched pages, and MCP results so it can serve them back later instead of re-running the work. Anything it writes to those caches is passed through a redactor first, so a credential that appeared in output does not sit on disk in plain text and does not get replayed into a later session. This is unconditional — there is no flag to turn it on, and it applies to cached Bash and Task output (including the command string itself, which is where an inline `--token=...` would otherwise land), fetched web content, MCP tool results and their labels, `compress-text`/`handoff` payloads, and the raw JSON disk cache. Recognized shapes: Anthropic, OpenAI, AWS, GitHub, Slack, Stripe, npm, and Google keys; JWTs; `Authorization: Bearer`/`Basic` headers; PEM private-key blocks; and generic `password=`/`secret=`/`api_key=` assignments in `.env`, connection-string, and query-string shape. A match is replaced by a `[REDACTED:<kind>]` marker naming which pattern fired.
 
 Two honest limits. It is a pattern matcher, not a classifier: a credential in a format it does not recognize — an internal token shape, a bare high-entropy string with no `key=` prefix — is cached as-is. And it protects what token-goat *stores*, not what your agent reads in real time; a secret printed to the terminal was already in the model's context before any caching happened. Treat it as damage control on the cache layer, not a reason to relax about printing secrets.