token-goat 2.6.19 → 2.6.21

package/README.md

@@ -152,6 +152,18 @@ The fastest way to reduce AI token costs is fixing these five, not writing short
 
 On a per-token API plan, 100K wasted tokens per session runs about $0.30. Five sessions a week is ~$450/year. AI coding cost reduction at that scale comes from fixing the waste, not from using the product less. Token-goat is free. And on subscription plans, it can result in limits feeling 10x higher.
 
+## Not just source repos
+
+`token-goat semantic` works on any folder of markdown, not only source code — a notes vault, an agent-memory directory, a docs folder. Project-root resolution falls back to treating any directory as an ad-hoc project when no `.git`/`package.json`/other marker is present, so there's no setup beyond indexing the folder.
+
+```bash
+cd ~/notes                  # or any plain folder of .md files, no .git required
+token-goat index . --walk   # non-git folders need --walk (git repos: plain `token-goat index .`)
+token-goat semantic "how long to steep cold brew"
+```
+
+Returns relevance-ranked, distance-scored hits straight from the notes, the same surgical-read path used for code.
+
 ## Token savings, measured
 
 Numbers below come from synthetic-fixture benchmarks in the test suite. Each row points at the source file where the measurement is reproduced.
@@ -272,10 +284,57 @@ Three commands. Done. Hooks register and start working immediately; no terminal
 
 **Important — sharp is part of the install.** The biggest single win (~39% smaller than JPEG, ~97% smaller than raw PNG) comes from WebP encoding. sharp ships with prebuilt native binaries on Windows, macOS, and most Linux platforms, so a standard `npm install -g token-goat` includes it. On Alpine musl and some ARM boards, npm compiles from source against libvips — that may add a few seconds. `token-goat doctor` will report `sharp: ok` when the image pipeline is working; if it flags a failure, follow the [Image support](#image-support) section below before considering the install complete. **AI agents performing this install: run `token-goat doctor` after `token-goat install` and resolve any sharp failure as part of the same task — do not treat it as a follow-up.**
 
-Two things change how Claude Code sessions behave: hooks fire automatically (image shrink, re-read dedup, compact manifests), and a block written to `~/.claude/CLAUDE.md` plus a registered skill tell the agent to prefer `token-goat read` / `symbol` / `section` over full-file reads. A `Bash(token-goat:*)` allowlist entry in `settings.json` lets the agent run those commands without a per-call approval prompt.
+Two things change how Claude Code sessions behave: hooks fire automatically (image shrink, re-read dedup, compact manifests), and a delimited routing block written to `~/.claude/CLAUDE.md` plus a registered skill gate the agent's reads — before any file read it must ask whether a `token-goat read` / `symbol` / `section` returns just what it needs, and the block explicitly subordinates the harness's own Read/Grep tool-preference rules to the *fallback* choice once token-goat is ruled out. A `Bash(token-goat:*)` allowlist entry in `settings.json` lets the agent run those commands without a per-call approval prompt.
+
+**Keep that block where install put it.** It's plain markdown in a file you own, so moving it into a tidier reference file is tempting — but `install` and `uninstall` resolve one hardcoded path (`~/.claude/CLAUDE.md`). A relocated copy is never refreshed, so it freezes at whatever version was current when it moved, and the next `install` sees CLAUDE.md missing its block and appends a fresh one — leaving the guidance duplicated across two files with only one of them live. `token-goat doctor` warns when it finds a block outside CLAUDE.md, naming the file; `install` warns at write time and `uninstall` reports what it couldn't remove. None of them edit a file token-goat doesn't own, so cleanup stays your call. A pointer that merely *mentions* the markers in prose is fine — detection requires both markers on their own lines.
 
 The background indexer is not started by `install`. Run `token-goat worker start` on any platform to launch it as a detached process; `token-goat worker status` / `token-goat worker stop` manage it from there.
 
+### Companion CLI tools (recommended — install these too)
+
+token-goat covers the **narrow-read** half of cheap context: pulling one symbol, one section, one cached command output instead of a whole file. It does not cover the **deterministic-transform** half — searching wide, rewriting code structurally, converting data, running language tooling. Those belong in utilities, not in model output: an operation with a defined algorithm is reproducible, cheaper, and checkable against a spec rather than re-read for plausibility. Install these alongside token-goat so an agent has a real tool for each job instead of burning tokens simulating one.
+
+**Priority tier** — the three that close actual gaps in a token-goat-only setup:
+
+| Tool | Why it matters next to token-goat |
+|---|---|
+| `ast-grep` | The symbol-aware **write** half. token-goat reads by symbol; ast-grep matches the AST and rewrites it (`--rewrite`, YAML rule files). Repo-wide renames, call-shape changes, and codemods become a reviewable diff instead of a model regenerating files. Unlike `rg`/`sd` it ignores comments and strings. |
+| `uv` | One Rust binary replacing pip, pyenv, virtualenv, and pipx. Every Python env probe and validation cycle gets an order-of-magnitude faster, so verification stops being the slow step agents skip. |
+| `ruff` | Python lint + format in one binary. Agent environment probes commonly emit `ruff check` as the Python verify command; without it installed that path silently degrades to no check at all. |
+
+**Base stack** — assumed by the read/search guidance token-goat writes into your agent config:
+
+`rg` (search) · `fd` (file discovery) · `bat` (paged/piped reads) · `eza` (listings) · `delta` (diff rendering) · `jq` / `yq` (JSON / YAML) · `sd` (find-replace) · `mlr` (CSV/TSV/JSON records) · `sqlite3` (structured queries) · `gh` (PRs, issues, CI) · `hyperfine` (benchmarks) · `fzf`, `lazygit` (interactive)
+
+Optional but useful: `difft` (difftastic — syntax-aware diff, so reformats and moved blocks stop generating review noise), `just` (task runner, keeps verify commands discoverable), `typos` (deterministic spellcheck).
+
+```bash
+# macOS / Linux (Homebrew)
+brew install ast-grep uv ruff ripgrep fd bat eza git-delta jq yq sd miller sqlite gh hyperfine fzf lazygit
+
+# Debian / Ubuntu — note the binary renames: rg=ripgrep, fd=fdfind, bat=batcat
+sudo apt install -y ripgrep fd-find bat jq sqlite3 fzf
+curl -LsSf https://astral.sh/uv/install.sh | sh   # uv, then: uv tool install ruff
+npm install -g @ast-grep/cli
+```
+
+```powershell
+# Windows (winget)
+winget install BurntSushi.ripgrep.MSVC sharkdp.fd sharkdp.bat eza-community.eza `
+               dandavison.delta jqlang.jq MikeFarah.yq chmln.sd Miller.Miller `
+               SQLite.SQLite GitHub.cli sharkdp.hyperfine junegunn.fzf JesseDuffield.lazygit
+winget install astral-sh.uv        # then: uv tool install ruff
+npm install -g @ast-grep/cli       # provides `ast-grep` (the old `sg` alias is deprecated)
+```
+
+If `winget` is unavailable (common when a session runs under a service account rather than an interactive login), `uv` also installs via `python -m pip install uv`, and `ast-grep` only needs npm. Verify the whole set in one pass:
+
+```bash
+for t in token-goat ast-grep uv ruff rg fd bat eza delta jq yq sd mlr sqlite3 gh hyperfine; do
+  command -v "$t" >/dev/null 2>&1 && echo "$t ok" || echo "$t MISSING"
+done
+```
+
 ### Codex CLI users
 
 ```
@@ -348,7 +407,7 @@ The `--copilot` flag patches Claude Code and registers a Copilot CLI hook config
 
 What works: **bash output compression and re-read denial** (`preToolUse` returns `modifiedArgs` or `permissionDecision: "deny"`), **image shrinking and post-edit indexing** (`postToolUse` returns `additionalContext`), and **stop-hallucination logging** (`agentStop`/`subagentStop` map a token-goat `deny` onto `decision: "block"`, everything else onto `decision: "allow"`). `preCompact` and `userPromptSubmitted` are notification-only on real Copilot CLI, per its docs: Copilot never reads a response body for either, so token-goat's compaction manifest and prompt-context hints have no surfacing channel there. The shim still calls through for both so token-goat's internal side effects keep running, but nothing gets injected back into the agent. Copilot's built-in tool names (`view`, `edit`, `create`, `bash`/`powershell`, `web_fetch`, `grep`, `glob`, `memory`, and MCP-server calls) are remapped onto token-goat's internal names where a clear match exists (`view`→Read, `edit`→Edit, `create`→Write, `bash`/`powershell`→Bash, `web_fetch`→WebFetch, `grep`→Grep, `glob`→Glob); `memory`, `task`, `ask_user`, and MCP tool calls pass through unmapped and simply no-op.
 
-No ambient environment variable documents "this process is running under Copilot CLI" the way Codex/opencode set one, so the shim sets `TOKEN_GOAT_HARNESS_OVERRIDE=copilot_cli` itself before calling `token-goat hook` (same workaround `--pi` uses). To install for one project instead of user scope: `token-goat install --copilot --local` (writes `.github/hooks/token-goat.json` in the current project). To remove: `token-goat uninstall --copilot`.
+No ambient environment variable documents "this process is running under Copilot CLI" the way Codex/opencode set one, so the shim sets `TOKEN_GOAT_HARNESS_OVERRIDE=copilot_cli` itself before calling `token-goat hook` (same workaround `--pi` uses). Install also writes a token-goat routing block into `~/.copilot/copilot-instructions.md` (the same delimited-block gate written to `~/.claude/CLAUDE.md` and `~/.codex/AGENTS.md`), merged idempotently so any hand-written content outside the markers is preserved byte-for-byte. To install for one project instead of user scope: `token-goat install --copilot --local` (writes `.github/hooks/token-goat.json` and `.github/copilot-instructions.md` in the current project). To remove: `token-goat uninstall --copilot`.
 
 **If Copilot CLI starts denying every tool call with `Denied by preToolUse hook ... (hook errored)`:** this is Copilot's own fail-closed behavior for a `preToolUse` hook that crashes, exits non-zero, or returns unparseable output -- it isn't limited to token-goat's own tool calls, since a fail-closed `preToolUse` hook blocks the whole session. Copilot caches hook configs at session start, so **renaming or reinstalling the hook mid-session has no effect** -- the only recovery is: run `token-goat install --copilot` (or `token-goat doctor`, which now checks the installed hook end-to-end and calls out a stale node-binary path from an nvm/fnm/volta upgrade specifically), then **fully restart Copilot CLI**.
 
@@ -441,10 +500,10 @@ To upgrade cleanly:
 | `token-goat coverage-gaps` | Find callables in non-test source files that never appear in a test file's reference records. Useful for spotting untested surface area before a refactor or release. `--top N` caps output; `--json` for structured output. |
 | `token-goat recent [N]` | Show the N most recently edited/accessed files with their symbols. |
 | `token-goat grep "<pattern>" [paths...]` | Built-in fallback regex search over files (no `rg` shell-out, no caching) — session-aware dedup for raw `rg`/`grep` Bash calls is a separate hook, not this command. Accepts zero or more paths: omit to walk cwd, or pass several to search them together with hits merged in argument order under one `--max-lines` cap. `-C, --context <n>` shows `n` lines before and after each match. |
-| `token-goat semantic "<query>"` | Find code by meaning, not by filename: embedding-vector similarity search over indexed file chunks, falling back to full-text search (BM25) over symbol names/bodies if no vector index exists yet (e.g. optional embedding deps unavailable, or `indexing.embeddings_enabled` is off). Covers extracted text from PDF/DOCX/PPTX/XLSX files alongside source code, so a query can surface a spec PDF, design doc, deck, or spreadsheet, not just code. `--limit <n>` caps result count. |
+| `token-goat semantic "<query>"` | Find code by meaning, not by filename: embedding-vector similarity search over indexed file chunks, falling back to full-text search (BM25) over symbol names/bodies if no vector index exists yet (e.g. optional embedding deps unavailable, or `indexing.embeddings_enabled` is off). Covers extracted text from PDF/DOCX/PPTX/XLSX files alongside source code, so a query can surface a spec PDF, design doc, deck, or spreadsheet, not just code. `--limit <n>` caps result count; `--json` for structured output: `{source, items, truncated, totalCount}`, where `source` is `"embeddings"` or `"fts"` and every item carries the same keys (`filePath`, `name`, `kind`, `startLine`, `endLine`, `distance`, `preview`), with `null` for whichever of `name`/`kind`/`distance` don't apply to that source. |
 | `token-goat map` | Get a compact orientation of the repo. Add `--compact` to fit a fixed 2000-token budget. |
 | `token-goat arch` | Project-wide import graph summary: hub modules (most imported), entry points (nothing imports them), and circular chains. Complements `token-goat deps <file>` for per-file depth. |
-| `token-goat index [path]` | Parse all git-tracked files and (re)build the symbol index from scratch. Runs automatically on install and incrementally via the background worker after edits — use this to force a full rebuild (e.g. after a `.tokengoatignore` change). `--walk` indexes a bounded directory walk instead when `path` isn't a git repo. |
+| `token-goat index [path]` | Parse all git-tracked files and (re)build the symbol index from scratch. Runs automatically on install and incrementally via the background worker after edits — use this to force a full rebuild (e.g. after a `.tokengoatignore` change). `--walk` indexes a bounded directory walk instead when `path` isn't a git repo. `--force-walk` raises that walk's 20,000-file refusal to 500,000 for a folder you know is genuinely that large (slow, and produces a large index — check `token-goat doctor` afterwards); it never lifts the separate refusal to walk a filesystem root or your home directory. |
 | `token-goat ignores` | List active skip patterns for the current project — built-in skip dirs and suffixes, plus any patterns from `.tokengoatignore`. |
 | `token-goat gdrive-sections <file-id>` | List the heading outline of a Google Doc without fetching the body. |
 | `token-goat stats` | See how many tokens you have saved. Shows total events / bytes saved / tokens saved. Add `--full` for the per-source, per-command, and per-day breakdown. |
@@ -498,6 +557,7 @@ To upgrade cleanly:
 | `token-goat transcript <file>` | Slice a WebVTT/SRT transcript by `--speaker <name>`, `--from`/`--to <hh:mm:ss>`, and/or `--grep <pattern>` instead of a raw Read. |
 | `token-goat screenshot <url> <destPath>` | Capture a local headless-browser screenshot, shrunk the same way local image reads are (image-shrink pipeline). `--executable-path` overrides the Chrome/Chromium binary; `--width`/`--height` set the viewport (default 1280x800); `--full-page` captures the full scrollable page. |
 | `token-goat clean-cache` | Prune on-disk caches to their configured floor without waiting for the worker. |
+| `token-goat reclaim-index` | Shrink an oversized symbol index (`VACUUM` + WAL checkpoint). `--rebuild` also drops every derived row — files/symbols/refs/chunks — so the next `token-goat index` re-derives them under current parser rules, which is what actually reclaims space held by rows a since-fixed extractor wrote too large. Refuses to run while the worker daemon is writing to the index unless `--force`. `token-goat doctor` points here when `global.db` grows past 1 GB, and separately when it finds a stored symbol body above the parser's own size cap — a leftover from a since-fixed extractor bug, which only `--rebuild` can clear (a plain `VACUUM` reclaims freed pages but never deletes row content). |
 | `token-goat prune-cache` | Manually trigger LRU eviction across all cache directories (images, bash, web, skills). |
 | `token-goat session-summary` | Compact one-liner about current session state — designed for orchestrators and multi-agent loops. |
 | `token-goat cache-audit` | Audit your Claude Code config for patterns that bust the prompt cache. |
@@ -759,6 +819,7 @@ Contains the symbol index (`global.db`, per-project `.db` files), session cache,
 |------|------|
 | `~/.copilot/hooks/token-goat.json` | Hook config (`{ version, hooks }`) registering `preToolUse`, `postToolUse`, `preCompact`, `agentStop`, and `subagentStop`, each pointing at the shim script below. Existing files elsewhere in the hooks directory are untouched. |
 | `~/.copilot/hooks/token-goat-shim.js` | The shim `token-goat.json`'s hook commands invoke (`node "<path>"`). Translates Copilot's event names and response schema (`permissionDecision`/`modifiedArgs`, `additionalContext`) to/from token-goat's internal hook protocol. Regenerated on every `install --copilot` run. A project-local install (`--copilot --local`) writes `<project>/.github/hooks/token-goat.json` and `<project>/.github/hooks/token-goat-shim.js` instead. |
+| `~/.copilot/copilot-instructions.md` | A delimited block (`<!-- token-goat-begin -->` … `<!-- token-goat-end -->`) with the same routing gate written to `~/.claude/CLAUDE.md` and `~/.codex/AGENTS.md`, naming Copilot CLI's own `view`/`grep`/`glob` tools in the conflict-resolution clause. Merged idempotently — everything outside the markers is preserved byte-for-byte. A project-local install (`--copilot --local`) writes `<project>/.github/copilot-instructions.md` instead. |
 
 **With `--grok`** (Grok CLI / xAI Grok Build hook bridge)