token-goat 2.9.4 → 2.9.6

package/README.md

@@ -11,7 +11,7 @@ permalink: /
 
 ***Give the model what it needs, not everything you have.***
 
-**85%** smaller reads · **97.4%** image compression · **180+** filter & interception rules · **94–99%** skill overhead cut · compaction memory · **prompt injection** guard · **3.7 GB** never reached the model · **1.1 Gt** tokens saved
+**85%** smaller reads · **49%** of first-read bytes withheld · **97.4%** image compression · **180+** filter & interception rules · **94–99%** skill overhead cut · compaction memory · **prompt injection** guard · **3.7 GB** never reached the model · **1.1 Gt** tokens saved
 
 **Reduces AI token use/costs by 40–90%, and improves its focus. Fully automated, always online.**
 
@@ -70,7 +70,8 @@ The fastest way to reduce AI token costs is fixing these five, not writing short
 | 3.3 MB screenshot lands in model context | 84 KB compressed copy, 97.4% smaller |
 | Agent re-reads files from earlier in the session | "Already read this" reminder with narrow slice suggestion |
 | Read tool asks for lines the session was already given | Answered with a pointer at the copy already delivered instead of the file, when the text this read would return matches the text already served for it whole line for whole line. Proof rather than a read count, so it also covers a file inside the recent-read protection window; a changed file, a wider range, and `reread_deny = false` all pass through |
-| Same unchanged file read again through the shell (`cat`, `head`, `tail`, `sed -n '1,40p'`) | Byte-identical repeat replaced with a one-line pointer instead of the whole body — 4,031 bytes to 189 on this project. Only fires when the two runs match exactly, so a changed file is untouched; full text stays available via `token-goat bash-output <id>` |
+| Same unchanged file read again through the shell (`cat`, `head`, `tail`, `sed -n '1,40p'`) | Byte-identical repeat replaced with a one-line pointer instead of the whole body: 4,031 bytes to 189 on this project. A later read wholly inside an earlier one collapses the same way |
+| A shell read of a file overlaps an earlier one without sitting inside it (`sed -n '100,140p'` after `sed -n '120,160p'`) | Only the already-shown stretch is withheld, replaced in place by one line naming its line numbers and the command that returns it; every line not shown before arrives untouched. 56% of the second read on a real pair over this project's README. Line numbers come from the command's own ranges, so a `tail`, whose start the command does not state, is left alone. Full text stays available via `token-goat bash-output <id>` |
 | Part of a file re-read under a different command (`head -40 F` then `sed -n '1,30p' F`) | Repeat replaced with a pointer when every line coming back was already served for that same file this session. Matched on the text, on whole-line boundaries, so a read that adds any new lines is left whole; the record is dropped when the file is edited or the conversation is compacted. A file read with the Read tool counts as served too, limited to the lines that read actually handed over, and a read that came back truncated counts for nothing |
 | Read tool asks for a range that overlaps one already delivered | Only the overlapping stretch is withheld, replaced by one line naming the line numbers and the recall command; the new lines come through with their own numbers and spacing, byte for byte. A stretch is kept whenever dropping it would save less than the line replacing it costs, and a result carrying anything the redactor would strip, a truncated result, or a file changed on disk all pass through whole. Set `elide_served_lines = false` to disable |
 | Shell command run with colour on (`git`, `npm`, `npx`, `token-goat`) | Terminal escape sequences removed, leaving byte-for-byte the same output with colour off: 13.2% smaller on a real `git diff` here, and about a quarter of the bytes on the calls it fires for. Nothing is summarized or withheld, so there is no recall pointer; it runs on failed commands too, and only when the escapes are worth a rewrite |
@@ -114,7 +115,7 @@ The fastest way to reduce AI token costs is fixing these five, not writing short
 | Index-only files (lockfiles, source maps, bundles) read on every session | Pre-Read suppression for read-only files (package-lock.json, *.map, dist/), skipped unless explicitly edited |
 | Large markdown file read in full (README.md, CHANGELOG.md, CLAUDE.md ≥8 KB) | Heading tree intercepted instead — H1–H3 with `#2`/`#3` disambiguation; `token-goat section` shortcuts listed for well-known files; post-edit injects a re-read suggestion rather than the full file |
 | Subagent reads a ≥30 KB markdown file whole, on its first look at it | Off by default. Set `subagent_markdown_first_read_deny = true` under `[hints]` to block that read and answer with the heading tree instead. Only fires in a subagent, only on a first, un-ranged read of a `.md`/`.mdx`/`.markdown` file with at least three headings; a read that already asks for a line range, and every main-session read, pass through untouched |
-| A source file is read whole for the first time, with no earlier copy to compare it against | Off by default. Set `fold_code_bodies = true` under `[hints]` to keep each function's declaration and its first eight lines, replacing the rest with one line naming the symbol, the exact range removed, and the command that returns it. Comment blocks of twelve lines or more fold to their first two on the same terms. Imports, signatures, type declarations, classes and interfaces are never folded, and a file whose index no longer matches what is on disk keeps its bodies. Measured on this repository's own source: 229 of 256 files fold, and 43% of what a read of them delivers is removed. A read that already names an offset or a limit, and any file holding something that looks like a secret, pass through untouched |
+| A source file is read whole for the first time, with no earlier copy to compare it against | Keeps each function's declaration and its first eight lines, replacing the rest with one line naming the symbol, the exact range removed, and the command that returns it. Set `fold_code_bodies = false` under `[hints]` to turn it off. Comment blocks of twelve lines or more fold to their first two on the same terms. Imports, signatures, type declarations, classes and interfaces are never folded, and a file whose index no longer matches what is on disk keeps its bodies. Measured on this repository's own source: 229 of 256 files fold, and 43% of what a read of them delivers is removed. A read that already names an offset or a limit, and any file holding something that looks like a secret, pass through untouched |
 | PDF opened via Read | Full read denied; PDF shows page count and outline (`token-goat pdf-extract` pulls the actual text, optionally paged/sliced, when the outline isn't enough) |
 | Excel/PowerPoint/Word file (.xlsx/.pptx/.docx) opened via Read | Full read denied; redirects to the matching narrow-slice command family (`xlsx-sheets`/`xlsx-head`/`xlsx-range`/`xlsx-query`, `pptx-outline`/`pptx-slide`/`pptx-notes`/`pptx-text`, `docx-outline`/`docx-text`) instead of extracting the whole document as text |
 | Other Office binary (.odt, .ods, .ott, .odp) opened via Read | Full read denied; redirects to `pandoc` for text extraction (no dedicated reader for these formats yet) |
@@ -185,10 +186,12 @@ Returns relevance-ranked, distance-scored hits straight from the notes, the same
 
 ## Token savings, measured
 
-Numbers below come from synthetic-fixture benchmarks in the test suite. Each row points at the source file where the measurement is reproduced.
+Numbers below come from synthetic-fixture benchmarks in the test suite, except the first two rows, which are measured on a corpus of real sessions and say so. Each row points at the source file where the measurement is reproduced.
 
 | Source | Improvement | Measured impact | Where |
 |--------|-------------|-----------------|-------|
+| Large-document outline | A read naming no `offset` or `limit` on a markdown document over 8 KB with six or more headings returns the lead-in and heading tree, plus the `token-goat section` command that reads any section verbatim | Measured on 13,870 real reads across 5,104 sessions: withholds 40.4% of all bytes the Read tool delivers. A 104,890 B document returns 1,541 B with its opening paragraph intact. Sections sum to the document, so a reader who pulls every one back pays about 3% over reading it whole | `src/hooks_read.ts` (`foldMarkdownOutline`) |
+| Large-source skeleton | A read naming no `offset` or `limit` on a source file over 12 KB with eight or more declarations returns the preamble and one line per declaration, each withheld run naming the `token-goat read "file::Symbol"` or ranged `Read` that returns it | Measured on the same corpus: withholds up to 8.6% of all bytes the Read tool delivers, a ceiling rather than a booked total, since the shipping gates are narrower than the measurement's. The skeleton is 10.5% of a `.ts` file. Median symbol body is 4.1% of its file, so a reader breaks even at roughly 22 bodies pulled back out of an average 30.8 | `src/hooks_read.ts` (`foldSourceSkeleton`) |
 | Image shrink | Every still is encoded both ways and the smaller file wins, rather than one codec being assumed | 2560x1440 screenshot-shaped PNG: 2,782,963 -> 259,475 bytes (91% smaller) | `src/image_shrink.ts` (codec selection) |
 | Repomap output | `--compact` trims the top-symbols list to 10 (vs 30) and drops the recent-files section and per-symbol locations | Denser overview for the same byte budget | `src/baseline.ts` (`buildProjectMap`, `token-goat map --compact`) |
 | DB reindex | Batched single transaction + composite indexes on `(file_id, kind)` | 100 files / 10K rows: 84 s → 1 s (~80× faster) | `src/parser.ts`, `src/db.ts` (index migration) |