token-goat 2.9.12 → 2.9.14

package/README.md

@@ -117,7 +117,11 @@ The fastest way to reduce AI token costs is fixing these five, not writing short
 | Subagent reads a ≥30 KB markdown file whole, on its first look at it | Off by default. Set `subagent_markdown_first_read_deny = true` under `[hints]` to block that read and answer with the heading tree instead. Only fires in a subagent, only on a first, un-ranged read of a `.md`/`.mdx`/`.markdown` file with at least three headings; a read that already asks for a line range, and every main-session read, pass through untouched |
 | A source file is read whole for the first time, with no earlier copy to compare it against | Keeps each function's declaration and its first eight lines, replacing the rest with one line naming the symbol, the exact range removed, and the command that returns it. Set `fold_code_bodies = false` under `[hints]` to turn it off. Comment blocks of twelve lines or more fold to their first two on the same terms. Imports, signatures, type declarations, classes and interfaces are never folded, and a file whose index no longer matches what is on disk keeps its bodies. Measured on this repository's own source: 229 of 256 files fold, and 43% of what a read of them delivers is removed. A read that already names an offset or a limit, and any file holding something that looks like a secret, pass through untouched |
 | PDF opened via Read | Full read denied; PDF shows page count and outline (`token-goat pdf-extract` pulls the actual text, optionally paged/sliced, when the outline isn't enough) |
-| Excel/PowerPoint/Word file (.xlsx/.pptx/.docx) opened via Read | Full read denied; redirects to the matching narrow-slice command family (`xlsx-sheets`/`xlsx-head`/`xlsx-range`/`xlsx-query`, `pptx-outline`/`pptx-slide`/`pptx-notes`/`pptx-text`, `docx-outline`/`docx-text`) instead of extracting the whole document as text |
+| Excel/PowerPoint/Word file (.xlsx/.pptx/.docx) opened via Read | Full read denied; redirects to the matching narrow-slice command family (`xlsx-sheets`/`xlsx-columns`/`xlsx-head`/`xlsx-range`/`xlsx-query`, `pptx-outline`/`pptx-slide`/`pptx-notes`/`pptx-text`, `docx-outline`/`docx-tables`/`docx-text`) instead of extracting the whole document as text |
+| SQLite database (.db, .sqlite, .sqlite3, .db3) opened via Read | Full read denied; redirects to the SQLite narrow-slice family (`sqlite-tables`/`sqlite-schema`/`sqlite-query`) instead of reading binary bytes into context |
+| Parquet file (.parquet) opened via Read | Full read denied; redirects to DuckDB for SQL querying instead of reading binary columnar data |
+| Large JSON or YAML file (≥20 KB) read in full | Top-level keys or array summary previewed; redirects to `json-outline`/`json-query` or `yaml-outline`/`yaml-query` instead of blowing context on multi-megabyte payloads |
+| Large JSON Lines file (.jsonl ≥20 KB) read in full | Record count and first-record schema shown; suggests slicing records with offset/limit instead of loading millions of tokens |
 | Other Office binary (.odt, .ods, .ott, .odp) opened via Read | Full read denied; redirects to `pandoc` for text extraction (no dedicated reader for these formats yet) |
 | Large CSV or TSV file (≥10 KB) read in full | Column headers, row count, and 3 sample rows shown; `token-goat csv-query` projects columns and/or filters rows instead of a full read; `duckdb` query suggestion for very large tabular data |
 | WebFetch returns a page's full raw HTML | HTML-to-text extraction strips markup/scripts/styles before the model ever sees it — readable prose instead of a wall of tags |
@@ -294,6 +298,15 @@ Built-in output compression covers 150+ dev tool CLIs: `pytest`, `jest` / `vites
 
 A failing `pytest` / `jest` / `vitest` / `go test` / `cargo test` run (including bare `npm test`/`npm run test`/`yarn test`/`pnpm test`) also gets a one-line advisory naming the exact `token-goat bash-output <id> | token-goat failures` command to run instead of re-reading the raw dump. Silent on a passing run, a non-test command, or output too small to be worth reducing.
 
+Give the command three ways: as positional arguments (`token-goat compress pytest -v tests/`), as one string with `-c` / `--cmd`, or base64-encoded with `--cmd-b64`. The base64 form preserves quotes, backslashes, and dollar signs that a harness or an intermediate shell would otherwise mangle; it takes precedence when more than one form is supplied. `--native` runs the command through the platform's own shell (`cmd.exe` on Windows) instead of bash, which keeps Windows path backslashes intact for tools that reject the forward-slash form.
+
+`-q` / `--quiet-success` trades a clean run's output for one line. On exit code 0 the full output is stored for recall and only the summary prints; on any other exit code the compressed output is delivered as usual.
+
+```
+$ token-goat compress -q --cmd "npm run lint"
+[tg: ok] npm run lint (22.2s, recall: token-goat bash-output 326d9800e732eb9d)
+```
+
 ### 6. Context pressure
 
 Token-goat tracks how close a session is to the autocompact trigger and tightens its hints as the window fills. Surgical-read suggestions kick in on progressively smaller files as pressure builds (500 lines at cool, down to 50 at critical), so large reads get flagged before they tip the session over. The PreCompact manifest also shrinks: capped at 500 tokens once the window runs hot, 300 once critical, so it stops contributing to the pressure it measures. The denominator is always the fixed 660,000-token autocompact trigger budget, not the model's raw context window, so the same thresholds apply across models. Run `token-goat doctor --context` to see the current footprint.
@@ -474,6 +487,8 @@ The file-by-file table for each harness, and the path that file sits at: **[What
 
 Hooks fire automatically on every tool call once installed — nothing to start or restart there. The background worker is a separate, manual step: `token-goat worker start` launches it as a detached process, `token-goat worker status` checks it, `token-goat worker stop` kills it. It restarts itself automatically if it crashes or gets killed while the machine is running — an edit hook checks its liveness and respawns it, rate-limited to about once every 5 minutes. It does not survive a reboot or logout, though; re-run `worker start` after either. `token-goat uninstall` removes the hook entries, `CLAUDE.md` block, and skill directory, but does not touch a running worker — stop it separately with `token-goat worker stop` if you no longer want it running.
 
+To move to a newer release, `token-goat upgrade` installs it from npm and then re-runs `token-goat install`, so your hooks and integration manifests point at the build that just landed. `token-goat upgrade --check` reports whether one is available without installing anything, and `--json` gives the same answer for a script. Both go quiet when `network.offline` is set, and `token-goat doctor` mentions an available update at the end of its report.
+
 ## Verify
 
 ```
@@ -519,6 +534,8 @@ The pipeline is written in TypeScript and decodes PNG, JPEG, GIF, and BMP itself
 
 The one thing the decoders will do is decline. A file whose header asks for more than 256 MB of pixel buffer, or that is in a variant token-goat cannot read exactly, is refused rather than guessed at, and the untouched original goes to the model. An untouched original is the right picture, so the only cost of a refusal is that the read is not shrunk.
 
+When `tesseract.js` is installed, text-heavy images can also have their text extracted via OCR (`token-goat image-text <file>`) to replace raw pixel data with plain text. English (`eng`) is always enabled by default, and users can combine one or more additional languages (`token-goat image-text <file> --lang "fra,spa"`). Language preferences persist across version updates when saved globally (`token-goat config set image_shrink.ocr_lang "fra,spa"`), per-project in `.token-goat.toml` (`[image_shrink] ocr_lang = "fra+spa"`), or via `TOKEN_GOAT_OCR_LANG`. 16 languages are supported (`eng`, `fra`, `spa`, `deu`, `ita`, `por`, `nld`, `pol`, `rus`, `tur`, `swe`, `ara`, `chi_sim`, `chi_tra`, `jpn`, `kor`), each cryptographically verified against pinned package checksums to protect against supply-chain tampering.
+
 ## Stats display
 
 `token-goat stats` uses 24-bit ANSI color and Unicode block characters for gradient bars, sparklines, and the activity heatmap. In the right terminal it renders sharply. In the wrong one you get broken characters, flat gray blocks, or a "rich is not installed" error.
@@ -695,6 +712,8 @@ Bug reports go to the same place. The most useful ones include:
 - The matching log line from `%LOCALAPPDATA%\dfk-helper\token-goat\logs\` on Windows or `~/.local/share/token-goat/logs/` on Linux/WSL
 - What you expected and what actually happened
 
+To audit an existing AI session for missed savings, hook friction, or optimization opportunities before opening an issue, see the [Session Mining Prompt](docs/PROMPT_SESSION_MINING.md).
+
 For private questions, commercial licensing, or anything you'd rather not post publicly, contact me at token-goat@dfkhelper.com.
 
 ## Disclaimer

package/dist/token-goat-chunk-2ESBO4IN.mjs

@@ -0,0 +1,209 @@
+import { createRequire as __cjsRequire } from 'node:module';
+const require = __cjsRequire(import.meta.url);
+import {
+  NotAnOfficeDocumentError,
+  assertOoxmlWithinDeadline,
+  collectElements,
+  collectTextRuns,
+  decodeZipEntry,
+  ooxmlPartBudget,
+  ooxmlWorkDeadline,
+  parseOoxmlPart,
+  readOoxmlZip,
+  sortNumberedParts
+} from "./token-goat-chunk-YHGTGG6K.mjs";
+import {
+  compileGuardedRegex
+} from "./token-goat-chunk-GMOUBOX4.mjs";
+import {
+  init_define_import_meta_env
+} from "./token-goat-chunk-A37V4PBF.mjs";
+
+// src/pptx_extract.ts
+init_define_import_meta_env();
+function shapePlaceholderType(shape) {
+  const sp = shape;
+  return sp["p:nvSpPr"]?.["p:nvPr"]?.["p:ph"]?.["@_type"];
+}
+function slideShapes(parsedSlide) {
+  return collectElements(parsedSlide, "p:sp");
+}
+function shapeText(shape) {
+  return collectTextRuns(shape, "a:t").join(" ").trim();
+}
+function tableRowBlocks(parsedSlide) {
+  const blocks = [];
+  for (const tbl of collectElements(parsedSlide, "a:tbl")) {
+    for (const row of collectElements(tbl, "a:tr")) {
+      const cellTexts = collectElements(row, "a:tc").map((cell) => collectTextRuns(cell, "a:t").join(" ").trim());
+      const rowText = cellTexts.join(" | ").trim();
+      if (rowText.length > 0) blocks.push(rowText);
+    }
+  }
+  return blocks;
+}
+async function slidePathsInPresentationOrder(entries, budget) {
+  const presXml = decodeZipEntry(entries, "ppt/presentation.xml", budget);
+  const relsXml = decodeZipEntry(entries, "ppt/_rels/presentation.xml.rels", budget);
+  if (presXml === null || relsXml === null) return null;
+  const presParsed = await parseOoxmlPart(presXml);
+  const relsParsed = await parseOoxmlPart(relsXml);
+  const ridToTarget = /* @__PURE__ */ new Map();
+  for (const rel of collectElements(relsParsed, "Relationship")) {
+    if (rel["@_Id"] !== void 0 && rel["@_Target"] !== void 0) {
+      ridToTarget.set(rel["@_Id"], rel["@_Target"]);
+    }
+  }
+  const ordered = [];
+  const seen = /* @__PURE__ */ new Set();
+  for (const sldId of collectElements(presParsed, "p:sldId")) {
+    const rid = sldId["@_r:id"];
+    if (rid === void 0) continue;
+    const target = ridToTarget.get(rid);
+    if (target === void 0) continue;
+    const normalized = target.startsWith("slides/") ? `ppt/${target}` : `ppt/slides/${target}`;
+    if (entries[normalized] === void 0 || seen.has(normalized)) continue;
+    seen.add(normalized);
+    ordered.push(normalized);
+  }
+  return ordered.length > 0 ? ordered : null;
+}
+async function listSlideParts(filePath) {
+  const entries = await readOoxmlZip(filePath, ".pptx");
+  const budget = ooxmlPartBudget();
+  const slidePaths = await slidePathsInPresentationOrder(entries, budget) ?? sortNumberedParts(
+    Object.keys(entries).filter((p) => /^ppt\/slides\/slide\d+\.xml$/.test(p)),
+    /slide(\d+)\.xml$/
+  );
+  if (slidePaths.length === 0) throw new NotAnOfficeDocumentError(`no slides found in ${filePath} (not a valid .pptx?)`);
+  return { entries, slidePaths, budget };
+}
+function relsPathFor(slidePath) {
+  const idx = slidePath.lastIndexOf("/");
+  return `${slidePath.slice(0, idx)}/_rels/${slidePath.slice(idx + 1)}.rels`;
+}
+function resolveRelativeTarget(basePath, target) {
+  const baseDir = basePath.slice(0, basePath.lastIndexOf("/"));
+  const segments = `${baseDir}/${target}`.split("/");
+  const resolved = [];
+  for (const segment of segments) {
+    if (segment === "" || segment === ".") continue;
+    if (segment === "..") resolved.pop();
+    else resolved.push(segment);
+  }
+  return resolved.join("/");
+}
+async function notesPathFor(entries, slidePath, budget) {
+  const relsXml = decodeZipEntry(entries, relsPathFor(slidePath), budget);
+  if (relsXml === null) return null;
+  const relsParsed = await parseOoxmlPart(relsXml);
+  for (const rel of collectElements(relsParsed, "Relationship")) {
+    if (rel["@_Target"] !== void 0 && rel["@_Type"]?.endsWith("/notesSlide") === true) {
+      return resolveRelativeTarget(slidePath, rel["@_Target"]);
+    }
+  }
+  return null;
+}
+async function parseSlide(entries, path, budget) {
+  const xml = decodeZipEntry(entries, path, budget);
+  if (xml === null) throw new Error(`missing part: ${path}`);
+  return parseOoxmlPart(xml);
+}
+async function notesTextFor(entries, notesPath, budget) {
+  if (notesPath === null) return "";
+  const xml = decodeZipEntry(entries, notesPath, budget);
+  if (xml === null) return "";
+  const parsed = await parseOoxmlPart(xml);
+  const shapes = collectElements(parsed, "p:sp");
+  const bodyShape = shapes.find((s) => shapePlaceholderType(s) === "body");
+  return bodyShape !== void 0 ? shapeText(bodyShape) : collectTextRuns(parsed, "a:t").join(" ").trim();
+}
+async function pptxOutline(filePath, deadline = ooxmlWorkDeadline()) {
+  const { entries, slidePaths, budget } = await listSlideParts(filePath);
+  const out = [];
+  for (let i = 0; i < slidePaths.length; i++) {
+    assertOoxmlWithinDeadline(deadline, "Narrow the read to specific slides with pptx-slide, or use a smaller deck.");
+    const path = slidePaths[i];
+    const parsed = await parseSlide(entries, path, budget);
+    const shapes = slideShapes(parsed);
+    const titleShape = shapes.find((s) => {
+      const t = shapePlaceholderType(s);
+      return t === "title" || t === "ctrTitle";
+    });
+    const title = titleShape !== void 0 ? shapeText(titleShape) : "";
+    const allText = collectTextRuns(parsed, "a:t").join(" ");
+    const bodyChars = Math.max(0, allText.length - title.length);
+    const hasNotes = (await notesTextFor(entries, await notesPathFor(entries, path, budget), budget)).length > 0;
+    out.push({ slide: i + 1, title, bodyChars, hasNotes });
+  }
+  return out;
+}
+async function slideTextFromParts(entries, slidePaths, slideNumber, includeNotes, budget) {
+  if (slideNumber < 1 || slideNumber > slidePaths.length) {
+    throw new Error(`slide ${slideNumber} out of range (this deck has ${slidePaths.length} slides)`);
+  }
+  const path = slidePaths[slideNumber - 1];
+  const parsed = await parseSlide(entries, path, budget);
+  const shapes = slideShapes(parsed);
+  const blocks = [...shapes.map(shapeText).filter((t) => t.length > 0), ...tableRowBlocks(parsed)];
+  const lines = [`# Slide ${slideNumber}`, ...blocks];
+  if (includeNotes) {
+    const notes = await notesTextFor(entries, await notesPathFor(entries, path, budget), budget);
+    if (notes.length > 0) lines.push("", "## Speaker notes", notes);
+  }
+  return lines.join("\n\n");
+}
+async function pptxSlideText(filePath, slideNumber, includeNotes) {
+  const { entries, slidePaths, budget } = await listSlideParts(filePath);
+  return slideTextFromParts(entries, slidePaths, slideNumber, includeNotes, budget);
+}
+async function pptxAllSlidesText(filePath, includeNotes, deadline = ooxmlWorkDeadline()) {
+  const { entries, slidePaths, budget } = await listSlideParts(filePath);
+  const slideTexts = [];
+  for (let i = 1; i <= slidePaths.length; i++) {
+    assertOoxmlWithinDeadline(deadline, "Narrow the read to specific slides with pptx-slide, or use a smaller deck.");
+    slideTexts.push(await slideTextFromParts(entries, slidePaths, i, includeNotes, budget));
+  }
+  return slideTexts.join("\n\n");
+}
+async function pptxNotesText(filePath, slideNumber, deadline = ooxmlWorkDeadline()) {
+  const { entries, slidePaths, budget } = await listSlideParts(filePath);
+  const targets = slideNumber !== void 0 ? [slideNumber] : slidePaths.map((_, i) => i + 1);
+  const sections = [];
+  for (const n of targets) {
+    assertOoxmlWithinDeadline(deadline, "Narrow the read to one slide with the slide-number argument, or use a smaller deck.");
+    if (n < 1 || n > slidePaths.length) throw new Error(`slide ${n} out of range (this deck has ${slidePaths.length} slides)`);
+    const notesPath = await notesPathFor(entries, slidePaths[n - 1], budget);
+    const text = await notesTextFor(entries, notesPath, budget);
+    if (text.length > 0) sections.push(`# Slide ${n} notes
+
+${text}`);
+  }
+  return sections.join("\n\n");
+}
+async function pptxTextGrep(filePath, pattern, deadline = ooxmlWorkDeadline()) {
+  const { entries, slidePaths, budget } = await listSlideParts(filePath);
+  const guarded = compileGuardedRegex(pattern, "i");
+  if (!guarded.ok) throw new Error(`invalid --grep pattern: ${pattern} -- the pattern ${guarded.reason}`);
+  const re = guarded.re;
+  const out = [];
+  for (let i = 0; i < slidePaths.length; i++) {
+    assertOoxmlWithinDeadline(deadline, "Narrow the read to specific slides with pptx-slide, or use a smaller deck.");
+    const parsed = await parseSlide(entries, slidePaths[i], budget);
+    const text = collectTextRuns(parsed, "a:t").join(" ");
+    if (re.test(text)) {
+      const idx = text.search(re);
+      const snippet = text.slice(Math.max(0, idx - 40), idx + 80).trim();
+      out.push({ slide: i + 1, snippet });
+    }
+  }
+  return out;
+}
+
+export {
+  pptxOutline,
+  pptxSlideText,
+  pptxAllSlidesText,
+  pptxNotesText,
+  pptxTextGrep
+};