pi-supernova 0.0.11 → 0.1.0

package/CHANGELOG.md

@@ -2,11 +2,82 @@
 
 ## [Unreleased]
 
+## [0.1.0] - 2026-09-04
+
+### Fixed
+
+- Each program uses a separate worker and transaction. Late callbacks, idle worker errors, concurrent runs, startup cancellation, and startup deadlines no longer share state.
+- Command deadlines stop descendants. Signal exits report failure. Native commits recover partial replacements; writes reject missing content and edits reject overlapping matches.
+- Unified patches handle zero-context insertions, empty files, final-newline changes, and CRLF. Reads see external changes; evidence search includes staged files.
+- OMP tool calls use the current session’s enabled, permission-aware registry. Discovery supplies complete schemas. Batch options, failure envelopes, scheduling, and call identifiers remain consistent.
+- Output limits cover aggregate batch text, markers, and spill footers. Details remain valid JSON; logs report truncation. DataView ranges and own `__proto__` values survive serialization.
+
+### Changed
+
+- Acorn parses supported function and body forms. Return hints ignore comments and nested callbacks. Invalid collection arguments report errors.
+- The result ledger checks text equality, retains explicit-read pins across aliases, expires source records, and accepts a zero window.
+- Terminal width checks use Unicode graphemes. Expanded cards wrap all bounded result and log text without a preview-line limit.
+- Pi 0.85.0 native adapters respect active-tool changes. Pi has no cross-extension execution API; the README states this limit.
+- Live checks cover Pi 0.85.0 and OMP 18.1.10, including expanded terminal output.
+
+## [0.0.15] - 2026-09-04
+
+### Removed
+
+- Dead code (−638 lines net): the static `extractOperationsFromCode` source-regex preview (the live trace superseded it), `SafeText`/`fitOutputLines`/`wrapPlainToWidth` (the call slot is always empty), `shutdownGuestWorkers`, `decode.js` helpers nobody imported, `vfs.clear`, the `tools` alias for `nova`, unused bridge methods (`hasExecutor`, `clearVfsCache`, `isMutating`), `test/verify-width-crash.mjs`, and unused status-header knobs (`spinnerFrame`, `iconOverride`).
+- `edit` no longer accepts a unified diff (`patch`/`oldText` starting with `@@`); `apply_patch` / `patch()` is the one way to apply a diff.
+- `nova.surface`/`nova.snap` host-side unwrapping and their dedicated RPC methods: both route through `nova.call` and the guest unwraps once.
+
+### Changed
+
+- One `relativeSlash`, one `isTestPath`, one declaration-span computation (`WorkspaceIndex.spansOf`, cached per file) instead of three copies each; search adapters (fuzzy find, grep, glob listing) live in `search.js`; internal-only names are no longer exported.
+- Render computes the card body once per width (was twice); fuzzy find stats only matched paths (was every file); results shorter than a collapsible run skip ledger hashing.
+
+### Fixed
+
+- `nova.speculate` end-to-end (an end-to-end test now covers rollback and commit through the worker RPC).
+
+## [0.0.14] - 2026-09-04
+
+### Added
+
+- **Seen-ledger.** The model's context window is treated as memory: a result never re-sends a run of lines (≥6, mostly substantive) that an earlier result in the session already contained. The run collapses to `⋯ N lines same as #12 · path:a–b ⋯`, citing the earlier program and, when the lines came from a file, the exact range, so one `read(path, a, n)` recovers them. Changed lines are never collapsed, so a re-read after an edit is exactly the delta. Lines the current program read with an explicit `offset`/`limit` are pinned and always shown. Programs are numbered (`ok #12 3ms`) to anchor the citations; the window is `seenWindow` programs (default 40) and resets on `session_start`. Not compression: every collapsed line already exists verbatim in the model's context. `/supernova` reports the session's returned vs. not-re-sent tokens.
+- **Edits close the loop.** `edit` returns the post-edit lines with numbers (±2 context, ≤40 lines) so no verification re-read is needed; a quick structural check (`check: unclosed '{' opened at line 1`, JSON parsed exactly; brackets/strings/templates/regex-aware, 0 false positives over 6,100 source files); and `X also referenced in a.js:12, b.js:40` for every declaration the edit changed, so callers are not forgotten.
+- **Failures carry their source.** A failing `bash` appends `--- source` with ±2 lines around each `path:line` it printed (≤4 sites), so a stack trace or test failure does not cost a read turn.
+- **Outlines carry relations.** Expanded spans in `read(path, {about})` end with `// used by: a.js:12, b.js:40`.
+
+Measured on a read→edit→verify→edit→verify→outline→outline loop: 20,967 tokens naive vs **3,098** (−85%); the verification re-read costs 49 tokens instead of 1,078.
+
+### Changed
+
+- README documents the research (Zero-Mem, Agent Zero Memory, Harness-of-Harness, SPACE) and the fff port precisely: which formulas and constants are fff's, what was not ported, and that the code was reimplemented in JavaScript after reading fff's Rust sources.
+
+## [0.0.13] - 2026-09-04
+
+### Fixed
+
+- A program that made no host calls rendered an empty frame (`╭─ nova: complete ─╮ / ╰──╯`); it is now a single status line. Expanded view still frames the returned value.
+
+## [0.0.12] - 2026-09-04
+
+### Added
+
+- `read(path, {about})`: one call returns the whole file as an outline: every declaration (including nested object/class methods) with its signature and line range, and only the bodies relevant to `about` expanded, with line numbers, under a character budget (default 8000, 6 spans, weak-match cutoff at 40% of the best span). A folded body reads `  75 function applyReplacements(target, content, edits) … 26 lines`, so the follow-up `read(path, 75, 26)` is known without another search. Across 5 files/questions: **65% fewer tokens** than reading the file (23,968 → 8,477), correct spans expanded.
+- fff (dmtrKovalenko/fff) ported to plain JS in `fuzzy.js`, no binary and no spawn:
+  - `glob`/`find` with free text (no glob characters) is a typo-tolerant, frecency-ranked path search: up to 2 skipped characters, smart-case, +40%/+20% exact/any filename bonus, frecency boost `base·f/100` with fff's AI-mode decay (3-day half-life, 7-day window, 30s…4h modification steps), +15% for git-modified files, directory-distance penalty from the last touched file.
+  - `grep` is smart-case, groups rows under one path header, lists files that *declare* the name first with declaration lines marked `*` (definition-first hinting), accepts `limit`, and falls back to a fuzzy line match when the literal has no hits (`CausualVfs` → `class CausalVfs`).
+  - The workspace index watches the tree with `fs.watch` (recursive) and refreshes on change instead of every 10s; TTL remains the fallback when watching is unavailable.
+- Structural surface detects indented methods (`async bash(params, signal) {`, `name: (a) => {`), so adapters and class members are their own spans for `evidence`, `snap`, and outlines.
+
+### Changed
+
+- Tool guidance: `evidence(question)` across the repo or `read(path, {about})` for one file; plain `read(path)` only for lines you will edit.
+
 ## [0.0.11] - 2026-09-04
 
 ### Added
 
-- `evidence(query, {k?, path?, maxChars?})` — zero-token evidence selection over the codebase after Zero-Mem (arXiv:2607.29377), implemented 1:1 with the paper's non-generative pipeline: declared spans are the context units and identifiers the entities (eq.3), entity–span weights `w(d,e)=c(e,d)/Σc` (eq.4), file→span→line hierarchy (eq.5, eq.11), a deterministic query profile and relational/local route (eq.6–7), lexical entity alignment and one IDF-damped co-occurrence propagation step (eq.8–9), personalized PageRank `π=(1−γ)r+γPᵀπ` over spans (eq.10, γ=0.85, 10 iterations, factored through the entity layer so it is O(nnz)), per-view min-max normalisation and ρ-weighted fusion (eq.12–13, ρ=0.7), closure with definition bridges and in-file neighbours (eq.14), and calibration that filters by boundary/answer type/lexical support and ranks by type compatibility (eq.15). Returns top-K (default 5, per the paper's Top-5 ≈ Top-10 finding) verbatim source spans with path and line provenance under a 6000-char budget. Across 8 understanding questions on this repo the correct span ranks first and the result costs **68% fewer tokens** than reading the files the question spans (43,916 → 14,147). Warm latency 1–4ms.
+- `evidence(query, {k?, path?, maxChars?})`: zero-token evidence selection over the codebase after Zero-Mem (arXiv:2607.29377), implemented 1:1 with the paper's non-generative pipeline: declared spans are the context units and identifiers the entities (eq.3), entity–span weights `w(d,e)=c(e,d)/Σc` (eq.4), file→span→line hierarchy (eq.5, eq.11), a deterministic query profile and relational/local route (eq.6–7), lexical entity alignment and one IDF-damped co-occurrence propagation step (eq.8–9), personalized PageRank `π=(1−γ)r+γPᵀπ` over spans (eq.10, γ=0.85, 10 iterations, factored through the entity layer so it is O(nnz)), per-view min-max normalisation and ρ-weighted fusion (eq.12–13, ρ=0.7), closure with definition bridges and in-file neighbours (eq.14), and calibration that filters by boundary/answer type/lexical support and ranks by type compatibility (eq.15). Returns top-K (default 5, per the paper's Top-5 ≈ Top-10 finding) verbatim source spans with path and line provenance under a 6000-char budget. Across 8 understanding questions on this repo the correct span ranks first and the result costs **68% fewer tokens** than reading the files the question spans (43,916 → 14,147). Warm latency 1–4ms.
 - Structural surface now records column-0 `const/let/var` bindings, so module-level tables are their own spans (also sharpens `snap`).
 
 ### Changed
@@ -37,7 +108,7 @@
 ### Changed
 
 - Result card redesigned as an aligned ledger: one row per call (status · tool · duration · `exit N` · `+a/-r` · target), no tree stems or spacer rows, durations humanized (`6.2s`), multi-line commands shown as their first line plus `…+N lines`, and paths fitted to width with the basename kept. Trace records now carry per-call `ms` and non-zero `exitCode`.
-- Guest programs run in a warm worker thread. A hard timeout or abort now terminates synchronous loops (`while (true) {}`), `process.exit()` only ends the program, and `maxHeapMb` (V8 `resourceLimits` plus a process-RSS watchdog for Bun) stops memory blow-ups — none of these can take the host down anymore. Warm-worker overhead is ~0.1ms per program and ~20µs per `nova.call`.
+- Guest programs run in a warm worker thread. A hard timeout or abort now terminates synchronous loops (`while (true) {}`), `process.exit()` only ends the program, and `maxHeapMb` (V8 `resourceLimits` plus a process-RSS watchdog for Bun) stops memory blow-ups. None of these can take the host down anymore. Warm-worker overhead is ~0.1ms per program and ~20µs per `nova.call`.
 - Result text is a compact JS literal (`ok 12ms` header, unquoted keys, no separator whitespace, one item per line only past 120 columns): ~43% fewer tokens than the previous pretty JSON. Logs appear under `--- logs` only when present.
 - `maxReturnChars` default lowered from 200000 to 32000.
 - Return values that JSON cannot express are rendered instead of collapsing to `[object Object]`: circular references, `Map`, `Set`, `BigInt`, `Error`, functions, typed arrays.
@@ -96,7 +167,7 @@
 
 ### Fixed
 
-- OMP no longer hangs on "Loading plugins…" — removed top-level import of `@oh-my-pi/pi-coding-agent/tui` from the extension (portable framed chrome only).
+- OMP no longer hangs on "Loading plugins…": removed top-level import of `@oh-my-pi/pi-coding-agent/tui` from the extension (portable framed chrome only).
 
 ## [0.0.3] - 2026-09-03
 

package/README.md

@@ -12,7 +12,13 @@ pi install npm:pi-supernova
 omp install npm:pi-supernova
 ```
 
-Load **before** other tool-owning packages so `registerTool` capture works. Restart the host after install.
+Restart the host after install. OMP uses the current session’s enabled tool registry, independent of extension load order.
+
+Pi 0.85.0 supplies tool schemas but has no cross-extension execution API. Supernova uses native adapters and executors registered through its API instance.
+
+An unrelated Pi extension’s tools are not callable through Supernova. OMP 18.1.10 supports those calls through its session registry.
+
+The runtime, transaction, patch, output-limit, and renderer fixes share the same code on both hosts. Disabled configured tools are not callable.
 
 ---
 
@@ -25,7 +31,11 @@ Load **before** other tool-owning packages so `registerTool` capture works. Rest
 | Parallel reads | Ad hoc | `callMany` Auto / `parallel()` |
 | Hosts | Separate packages | Same tarball for Pi **and** OMP |
 
-Guest code runs as an `AsyncFunction` in a worker thread (same trust class as host `bash`) — not an OS/VM sandbox. Tool calls go through `nova.call` RPC to the host thread; a hard timeout, abort, `process.exit`, or a memory blow-up terminates the worker without touching the harness.
+Guest code runs as an `AsyncFunction` in a new worker for each program. It has the same trust level as host `bash`.
+
+The host parses the bounded source with Acorn before execution. Tool calls use RPC. Deadlines and cancellation cover worker startup, tool discovery, and execution.
+
+A completed worker cannot supply callbacks or globals to another program. Concurrent programs have separate transactions, traces, cancellation signals, and call budgets.
 
 ---
 
@@ -46,13 +56,17 @@ async () => {
 }
 ```
 
-Globals: `nova` / `tools`, `parallel`, `pipeline`, `console`, plus shorthand `read` (path or path array), `write`, `edit`, `patch`, `evidence`, `surface`, `snap`, `bash`, and `exec`.
+Globals: `nova`, `parallel`, `pipeline`, `console`, plus shorthand `read` (path or path array), `write`, `edit`, `patch`, `evidence`, `surface`, `snap`, `bash`, and `exec`.
+
+Read discipline that keeps context small: `evidence(question)` across the repo, or `read(path, {about: question})` for one file (full structure, only relevant bodies expanded, ~65% fewer tokens than the file) → `read(path, offset, limit)` only for the lines you will edit.
 
-Read discipline that keeps context small: `surface(path)` (names only) → `evidence(question)` (the spans that answer it) → `read(path, offset, limit)` only for the lines you will edit.
+Results never repeat what the model already saw: a run of lines from an earlier result collapses to `⋯ 23 lines same as #12 · host-bridge.js:40–62 ⋯`; changed lines always show; `read(path, a, n)` shows a cited range again. `edit` returns the post-edit lines, a structural check, and who else references a changed declaration; a failing `bash` attaches the source behind the `path:line` it printed. A read→edit→verify loop costs ~15% of the naive token count.
 
-The returned value is rendered as a compact JS literal (unquoted keys, one item per line only when a container exceeds 120 columns) and capped at `maxReturnChars`. Strings are returned raw. This costs ~43% fewer tokens than pretty JSON — return small shaped values, not raw file dumps.
+File search is an in-process port of [fff](https://github.com/dmtrKovalenko/fff): `glob("hostbrdge")` finds `host-bridge.js` (typo-tolerant, frecency- and git-status-ranked), `grep` is smart-case with declaration lines first and a fuzzy fallback. None of it spawns a process.
 
-Unified Pi/OMP card — one aligned row per call (status · tool · duration · target) with bounded mutation diffs:
+The returned value is rendered as a compact JS literal (unquoted keys, one item per line only when a container exceeds 120 columns) and capped at `maxReturnChars`. Strings are returned raw. This costs ~43% fewer tokens than pretty JSON. Return small shaped values, not raw file dumps.
+
+Unified Pi/OMP card: one aligned row per call (status · tool · duration · target) with bounded mutation diffs.
 
 ```text
 ╭─── nova: 4 calls · 6.9s ─────────────────────────────────────╮
@@ -65,7 +79,9 @@ Unified Pi/OMP card — one aligned row per call (status · tool · duration ·
 ╰───────────────────────────────────────────────────────────────╯
 ```
 
-Multi-line commands show their first line plus a hidden-line count. Press Enter for a larger hunk budget, the full returned value, and logs.
+Multi-line commands show their first line and a hidden-line count. Expand the card to show the complete bounded return value and logs.
+
+Expanded text wraps at the terminal width without a preview-line limit. Operation lists and mutation diffs keep their separate 24-row budgets.
 
 ---
 
@@ -74,16 +90,23 @@ Multi-line commands show their first line plus a hidden-line count. Press Enter
 | API | Role |
 |-----|------|
 | `nova.search(query, limit?)` | Thin catalog hits |
-| `nova.describe(name)` | Parameter summary on demand |
+| `nova.describe(name)` | Complete JSON input schema on demand; explicit failure when no usable schema is available |
 | `nova.call(name, args)` | Host tool or native adapter |
-| `nova.callMany([{name,args}])` | Auto parallel wave — iterable array with `.mode` / `.results` |
-| `nova.evidence(query, {k?, path?, maxChars?})` | Top-K source spans (path, lines, verbatim text) that answer a question — zero-token evidence selection after Zero-Mem; ~68% fewer tokens than reading the files |
+| `nova.callMany([{name,args}])` | Auto parallel wave; iterable array with `.mode` / `.results` |
+| `nova.evidence(query, {k?, path?, maxChars?})` | Top-K source spans (path, lines, verbatim text) that answer a question. Zero-token evidence selection after Zero-Mem; ~68% fewer tokens than reading the files |
+| `read(path, {about})` | Whole-file outline with only the relevant bodies expanded; folded bodies show `line … N lines` |
 | `nova.surface(path)` | Structural outline for a source file |
 | `nova.snap(query, searchRoot?)` | Defining file (workspace-relative), line, signature, confidence, and context for a concept; served from the in-process index in well under 1ms |
 | `nova.has(name)` | Whether a catalog or native tool is callable (sync) |
-| `parallel(thunks)` / `pipeline(items, …stages)` | Raw `Promise.all` helpers |
+| `parallel(thunks)` / `pipeline(items, …stages)` | Array-based helpers; pipeline stages must be functions |
 | `nova.speculate(fn)` | Counterfactual branch (rollback / commit) |
 
+`nova.call` returns an explicit `{ok, value}` envelope. Convenience helpers throw when a host tool reports failure.
+
+`callMany` runs known read-only tools concurrently. Unknown tools and mutating actions run in order, including LSP rename operations.
+
+`nova.describe` preserves required fields, unions, enums, nested objects, and numeric constraints. OMP ArkType and Zod schemas convert to JSON Schema.
+
 Root Snap searches ignore hidden files. Passing a hidden search root includes hidden files beneath that root; Git metadata is always excluded.
 
 ---
@@ -101,16 +124,23 @@ Optional `~/.pi/agent/supernova.json` or `~/.omp/agent/supernova.json`
   "maxCallResultChars": 65536,
   "maxReturnChars": 32000,
   "maxHeapMb": 512,
+  "seenWindow": 40,
   "maxSearchResults": 12,
   "spillDir": null
 }
 ```
 
-`maxHeapMb` caps the guest worker heap (V8 `resourceLimits` on Node) and arms a process-RSS watchdog that terminates a runaway program on both Node and Bun.
+`maxCallResultChars` bounds text per call. A batch shares one text budget across all items; it does not repeat a joined copy.
 
-Defaults also set `excludeTools` (includes `supernova` and DCE helpers). An empty `"excludeTools": []` **replaces** those defaults — omit the key unless you mean that.
+Envelope fields and JSON encoding add overhead. Details have a separate 2,000-character JSON budget. Truncation flags cover returns, raw objects, and logs.
 
-Slash: `/supernova` — catalog size + captured executors.
+An optional `spillDir` retains complete truncated text. Inline spill footers and truncation markers fit within the configured text limit.
+
+`seenWindow` is how many programs back the seen-ledger remembers (set 0 to disable collapsing). `maxHeapMb` caps the guest worker heap (V8 `resourceLimits` on Node) and arms a process-RSS watchdog that terminates a runaway program on both Node and Bun.
+
+Defaults also set `excludeTools` (includes `supernova` and DCE helpers). An empty `"excludeTools": []` **replaces** those defaults, so omit the key unless you mean that.
+
+Slash command `/supernova`: callable catalog size, external tools, native adapters, and session token stats.
 
 ---
 
@@ -135,20 +165,41 @@ Pair with DCE last if you use it: `omp install npm:pi-deferred-context-engine`.
 
 | Symptom | Fix |
 |---------|-----|
-| `unknown tool "…"` | Follow the `Did you mean` hint, or `nova.search("")`; for host tools install supernova **first**; restart; `/supernova` |
-| `Rendered line exceeds terminal width` | ≥0.0.1 and restart so `render.js` reloads |
-| `callMany` / not iterable | ≥0.0.1 — return is an array with `.mode` / `.results` |
+| `unknown tool "…"` | Use `nova.search("")` and `/supernova`. Enable the tool in the current session. Restart after package changes. |
+| `Rendered line exceeds terminal width` | Install the current package and restart so the Unicode width code reloads. |
+| `callMany` / not iterable | ≥0.0.1; the return is an array with `.mode` / `.results` |
 | Extension missing on OMP | `omp install npm:pi-supernova` (needs `"omp".extensions`) |
 
 ---
 
 ## Limitations
 
-- Guest JS is **unsandboxed**. Adapter path jails are not a boundary against `import("node:fs")`. The worker only contains hangs, exits, and memory — not intent.
+- Guest JS is **unsandboxed**. Adapter path jails are not a boundary against `import("node:fs")`. The worker only contains hangs, exits, and memory, not intent.
 - Guest error messages carry `(line:col)` on Node; Bun's engine does not expose guest-relative positions.
-- `bash` / mutating tools flush speculative writes (transaction barrier); error rollback cannot undo that.
-- The workspace index refreshes its file list every 10s or on any supernova mutation; a file created by an external process can take up to 10s to appear in `glob`/`snap` (`read` is never stale).
-- Pre-1.0 package — APIs and TUI may still evolve between minor releases.
+
+## Transactions and file freshness
+
+- `bash` and mutating host tools commit pending writes before execution. Later rollback cannot undo those changes or external effects.
+- Native commits stage replacements and backups before installation. A commit failure restores earlier replacements; a recovery failure reports retained backup paths.
+- A nested `nova.speculate` branch cannot call external mutators. Await each branch before returning.
+- Native reads use current disk content unless a staged write replaces it. Evidence search includes new staged files.
+- Workspace file-list updates use filesystem watchers. Without a working watcher, external new files can take 10 seconds to appear in indexed searches.
+
+This is a pre-1.0 package. APIs and the terminal display can change between minor releases.
+
+## Research and prior art
+
+Supernova's retrieval and result shaping implement published methods. Where a paper's mechanism needs a model call it stays out of the tool; only the deterministic parts are implemented, and each is cited at the code that implements it.
+
+| Work | What we use it for | Where |
+|------|--------------------|-------|
+| **Zero-Mem: Zero-Token Memory Operations for LLM Agents**, Xiao, Zhu, Zhang, Chen, Hong, Zhuang, Zhang, Chen, Ouyang, Ren, Huang (arXiv:2607.29377) | `evidence(query)`: entity–context graph with co-occurrence weights (eq. 3–4), turn/window/episode hierarchy as line/span/file (eq. 5, 11), query profile and relational/local routing (eq. 6–7), lexical entity alignment and one propagation step (eq. 8–9), personalized PageRank over spans (eq. 10), per-view normalisation and ρ-weighted fusion (eq. 12–13), closure with bridges and neighbours (eq. 14), deterministic calibration (eq. 15). Top-K = 5 follows the paper's Top-5 ≈ Top-10 finding. | `evidence.js` |
+| **Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents**, Zhu, Wu (arXiv:2608.29606) | Every returned unit carries provenance (path, line range, verbatim text); the L0→L1→L2 read discipline (`surface` → `evidence`/`read(path, {about})` → `read(path, offset, limit)`); the citation-lock idea that a model should only cite what it actually opened. | `evidence.js`, `outline.js`, tool guidance |
+| **Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement**, Yan, Su, et al. (arXiv:2609.01481) | Progressive disclosure (index first, detail on demand) and carrying evidence forward instead of reconstructing it from code. | outline / result shaping |
+| **Act More, Decide Less: Skill-Guided Adaptive Action Chunking for Long-Horizon LLM Agents**, Yang, Jin, Zhao, et al. (arXiv:2609.02042) | Framing: one supernova program is an action chunk (one model decision, many primitive actions, stop at the first failing one). | runtime design |
+| **fff**, Dmitriy Kovalenko, MIT, <https://github.com/dmtrKovalenko/fff> | File search. We reimplemented fff's ranking in plain JavaScript after reading its Rust sources (`crates/fff-core/src/score.rs`, `dbs/frecency.rs`, `path_utils.rs`); the formulas and constants are fff's, the code is ours, and nothing runs out of process. Ported: typo-tolerant fuzzy path matching with boundary/consecutive/case bonuses and smart-case; exact-filename +40% and filename +20% bonuses; frecency boost `base·f/100` with fff's AI-mode decay (3-day half-life, 7-day window) and modification-recency steps (30s/5m/15m/1h/4h); git-modified +15%; directory-distance penalty from the current file (−1 per hop, floor −20); definition-first result hinting; fuzzy fallback on zero literal matches; weak-match cutoff; watcher-driven index refresh. Not ported: fff's SIMD/frizbee matcher (ours is an fzf-style greedy match with backward tightening), LMDB persistence (frecency is per session), and the MCP/Neovim surfaces. | `fuzzy.js`, `repo-index.js`, `host-bridge.js` |
+
+fff is © Dmitriy Kovalenko and contributors, released under the MIT License; this package is also MIT. If you install fff's own Pi extension (`@ff-labs/pi-fff`) alongside supernova, its `ffgrep`/`fffind` tools are captured and callable through `nova.call` like any other host tool.
 
 ## License
 

package/bottleneck.js

@@ -4,120 +4,121 @@ import { randomUUID } from "node:crypto";
 import { isString, isObject } from "./decode.js";
 import { truncateChars, formatValue } from "./format.js";
 
-export function serializeBounded(value, maxChars, label = "value") {
-  let serialized;
-  try {
-    serialized = isString(value) ? value : JSON.stringify(value);
-  } catch {
-    serialized = String(value);
-  }
-  if (serialized === undefined) serialized = "null";
-  return truncateChars(serialized, maxChars, label);
+function json(value) {
+  try { return JSON.stringify(value) ?? "null"; } catch { return JSON.stringify(String(value)); }
 }
-
-function extractContentText(content) {
-  if (!Array.isArray(content)) return "";
-  return content
-    .filter((part) => part && isObject(part) && part.type === "text" && isString(part.text))
-    .map((part) => part.text)
-    .join("\n");
+function detailsOf(raw) {
+  const details = raw?.details;
+  if (!isString(details)) return details;
+  try { return JSON.parse(details); } catch { return details; }
 }
-
-function extractRawString(raw, maxChars) {
+export function hostResultFailed(raw) {
+  const details = detailsOf(raw);
+  return raw?.isError === true || details?.ok === false || (Number.isInteger(details?.exitCode) && details.exitCode !== 0);
+}
+function extractRawString(raw) {
   if (raw == null) return "";
   if (isString(raw)) return raw;
   if (!isObject(raw)) return String(raw);
-  if (Array.isArray(raw.content)) return extractContentText(raw.content);
+  if (Array.isArray(raw.content)) return raw.content.filter(part => part?.type === "text" && isString(part.text)).map(part => part.text).join("\n");
   if (isString(raw.text)) return raw.text;
-  return serializeBounded(raw, maxChars, "host-result").text;
-}
-
-function summarizeDetails(details, maxChars) {
-  return serializeBounded(details, maxChars, "details").text;
+  return json(raw);
 }
 
-function maybeSpill(cappedText, fullText, config) {
-  if (fullText.length <= (config.maxCallResultChars ?? 65536)) return null;
-  if (!isString(config.spillDir) || config.spillDir.length === 0) return null;
-  try {
-    const dir = config.spillDir;
-    fs.mkdirSync(dir, { recursive: true, mode: 0o700 });
-    const filePath = path.join(dir, `${Date.now()}-${randomUUID().slice(0, 8)}.txt`);
-    fs.writeFileSync(filePath, fullText, { encoding: "utf8", mode: 0o600 });
-    return { pointer: filePath };
-  } catch {
-    return null;
-  }
-}
-
-function batchItems(details, maxChars) {
-  if (!isObject(details) || details.batch !== true) return undefined;
-  if (!Array.isArray(details.items)) return undefined;
-  return details.items.map((item) => truncateChars(item, maxChars, "host-result").text);
+/** Bound JSON before serialization; preserve small scalar fields such as exitCode. */
+function summarizeDetails(value, budget = 2000) {
+  const encoded = json(value);
+  if (encoded.length <= budget) return encoded;
+  const snapshot = JSON.parse(encoded);
+  const fit = (input, limit) => {
+    const serialized = json(input);
+    if (serialized.length <= limit) return input;
+    if (isString(input)) {
+      let low = 0;
+      let high = Math.min(input.length, limit);
+      while (low < high) {
+        const mid = Math.ceil((low + high) / 2);
+        if (json(truncateChars(input, mid).text).length <= limit) low = mid;
+        else high = mid - 1;
+      }
+      return truncateChars(input, low).text;
+    }
+    if (!isObject(input) && !Array.isArray(input)) return null;
+    const out = Array.isArray(input) ? [] : { truncated: true };
+    const entries = Object.entries(input);
+    if (!Array.isArray(input)) entries.sort((a, b) => json(a[1]).length - json(b[1]).length);
+    for (const [key, child] of entries) {
+      const used = json(out).length;
+      const overhead = Array.isArray(out) ? 1 : json(key).length + 2;
+      const available = limit - used - overhead;
+      if (available < 4) break;
+      const bounded = fit(child, available);
+      if (Array.isArray(out)) out.push(bounded);
+      else Object.defineProperty(out, key, { value: bounded, enumerable: true, configurable: true });
+      if (json(out).length > limit) {
+        if (Array.isArray(out)) out.pop();
+        else delete out[key];
+      }
+    }
+    return out;
+  };
+  return json(fit(snapshot, budget));
 }
 
-function spillSuffix(capped, text, config) {
-  const spill = maybeSpill(capped.text, text, config);
-  if (!spill?.pointer) return { spill, value: capped.text };
-  return { spill, value: `${capped.text}\n\n[full output spilled to ${spill.pointer}]` };
+function spill(fullText, config) {
+  if (!isString(config.spillDir) || !config.spillDir) return undefined;
+  fs.mkdirSync(config.spillDir, { recursive: true, mode: 0o700 });
+  const file = path.join(config.spillDir, Date.now() + "-" + randomUUID().slice(0, 8) + ".txt");
+  fs.writeFileSync(file, fullText, { encoding: "utf8", mode: 0o600, flag: "wx" });
+  return file;
 }
 
 export function packageHostResult(raw, config) {
   const maxChars = config.maxCallResultChars ?? 65536;
-  const isError = isObject(raw) && raw.isError === true;
-  const details = isObject(raw) ? raw.details : undefined;
-  const upstreamTruncated = isObject(details) && details.outputTruncated === true;
-  const text = extractRawString(raw, maxChars);
-  const items = batchItems(details, maxChars);
-  // Batch items travel as their own field; keep the details summary small and parseable.
-  const summarizedDetails =
-    details === undefined ? undefined : summarizeDetails(items ? { ...details, items: undefined } : details, 2000);
-
+  const details = detailsOf(raw);
+  const batch = details?.batch === true && Array.isArray(details.items) ? details.items : undefined;
+  const text = batch ? "" : extractRawString(raw);
   const capped = truncateChars(text, maxChars, "host-result");
-  if (!capped.truncated) {
-    return {
-      ok: !isError,
-      value: capped.text,
-      truncated: upstreamTruncated,
-      details: summarizedDetails,
-      items,
-    };
+  let truncated = capped.truncated || details?.outputTruncated === true;
+  const result = { ok: !hostResultFailed(raw), value: capped.text, truncated };
+  if (details !== undefined) result.details = summarizeDetails(batch ? { ...details, items: undefined } : details);
+  if (batch) {
+    let remaining = maxChars;
+    result.items = batch.map((item, index) => {
+      const bounded = truncateChars(item, Math.floor(remaining / (batch.length - index)), "host-result");
+      remaining -= bounded.text.length;
+      truncated ||= bounded.truncated;
+      return bounded.text;
+    });
   }
-
-  const { spill, value } = spillSuffix(capped, text, config);
-
-  return {
-    ok: !isError,
-    value,
-    truncated: true,
-    originalChars: capped.originalChars,
-    spill: spill?.pointer,
-    details: summarizedDetails,
-    items,
-  };
-}
-
-function clipLogs(logs, config) {
-  const logLines = Array.isArray(logs) ? logs : [];
-  const maxLogLines = config.maxLogLines ?? 100;
-  const maxLogLineChars = config.maxLogLineChars ?? 4096;
-  const clippedLogs = logLines.slice(0, maxLogLines).map((line) => {
-    const s = isString(line) ? line : String(line);
-    if (s.length <= maxLogLineChars) return s;
-    return s.slice(0, maxLogLineChars) + "…";
-  });
-  return { clippedLogs, logTruncated: logLines.length > maxLogLines };
+  result.truncated = truncated;
+  if (truncated) {
+    result.originalChars = batch ? batch.reduce((sum, item) => sum + String(item).length, 0) : text.length;
+    if (config.spillDir) {
+      const pointer = spill(batch ? batch.join("\n---\n") : text, config);
+      if (pointer) {
+        result.spill = pointer;
+        if (!batch) {
+          const footer = "\n[full output spilled to " + pointer + "]";
+          result.value = footer.length <= maxChars
+            ? truncateChars(text, maxChars - footer.length, "host-result").text + footer
+            : capped.text;
+        }
+      }
+    }
+  }
+  return result;
 }
 
 export function packageFinalReturn(value, logs, config) {
-  const maxReturn = config.maxReturnChars ?? 32000;
-  const serialized = truncateChars(isString(value) ? value.replace(/\n+$/, "") : formatValue(value), maxReturn, "return");
-  const { clippedLogs, logTruncated } = clipLogs(logs, config);
-  return {
-    returnValue: serialized.truncated ? serialized.text : value,
-    returnText: serialized.text,
-    returnTruncated: serialized.truncated,
-    logs: clippedLogs,
-    logTruncated,
-  };
+  const serialized = truncateChars(isString(value) ? value : formatValue(value), config.maxReturnChars ?? 32000, "return");
+  const maxLines = config.maxLogLines ?? 100;
+  let logTruncated = logs.length > maxLines;
+  const clipped = logs.slice(0, maxLines).map(line => {
+    const result = truncateChars(line, config.maxLogLineChars ?? 4096, "log");
+    logTruncated ||= result.truncated;
+    return result.text;
+  });
+  return { returnValue: serialized.truncated ? serialized.text : value, returnText: serialized.text,
+    returnTruncated: serialized.truncated, logs: clipped, logTruncated };
 }

package/catalog.js

@@ -1,15 +1,16 @@
 
 import { isString, isObject } from "./decode.js";
 
-export const NATIVE_TOOL_DEFINITIONS = [
+const NATIVE_TOOL_DEFINITIONS = [
   {
     name: "read",
     description: "Read UTF-8 workspace files by path, or resolve a concept query to source. Supports path arrays, offset, and limit.",
     parameters: { type: "object", properties: {
-      path: { type: "string", description: "Workspace-relative file path or concept query" },
+      path: { anyOf: [{ type: "string" }, { type: "array", items: { type: "string" } }], description: "Workspace-relative file path, concept query, or array of paths" },
       target: { anyOf: [{ type: "string" }, { type: "array" }], description: "File path/query or array of paths" },
       offset: { type: "number", description: "One-based starting line" },
       limit: { type: "number", description: "Maximum lines to return" },
+      about: { type: "string", description: "Question or symbol: returns the whole file as an outline with only the relevant bodies expanded" },
     } },
   },
   {
@@ -17,9 +18,9 @@ export const NATIVE_TOOL_DEFINITIONS = [
     parameters: { type: "object", properties: { path: { type: "string" }, content: { type: "string" } }, required: ["path", "content"] },
   },
   {
-    name: "edit", description: "Apply unique text replacements, or a unified diff, to a workspace file.",
+    name: "edit", description: "Apply unique text replacements to a workspace file; returns the post-edit lines, a structural check, and references to changed declarations.",
     parameters: { type: "object", properties: {
-      path: { type: "string" }, oldText: { type: "string" }, newText: { type: "string" }, edits: { type: "array" }, patch: { type: "string" },
+      path: { type: "string" }, oldText: { type: "string" }, newText: { type: "string" }, edits: { type: "array", description: "[{oldText, newText}] for several replacements in one call" },
     }, required: ["path"] },
   },
   {
@@ -51,13 +52,13 @@ export const NATIVE_TOOL_DEFINITIONS = [
     parameters: { type: "object", properties: { command: { type: "string" }, cwd: { type: "string" }, timeoutMs: { type: "number" } }, required: ["command"] },
   },
   {
-    name: "grep", description: "Search workspace file contents by pattern.",
+    name: "grep", description: "Search file contents. Smart-case regex; definition lines first (marked *); fuzzy fallback when nothing matches literally.",
     parameters: { type: "object", properties: {
-      pattern: { type: "string" }, path: { type: "string" }, glob: { type: "string" }, caseSensitive: { type: "boolean" },
+      pattern: { type: "string" }, path: { type: "string" }, glob: { type: "string" }, caseSensitive: { type: "boolean" }, limit: { type: "number" },
     }, required: ["pattern"] },
   },
   {
-    name: "glob", description: "List workspace files matching a glob pattern.",
+    name: "glob", description: "Find files: a glob pattern, or free text for typo-tolerant, frecency-ranked path search.",
     parameters: { type: "object", properties: { pattern: { type: "string" } }, required: ["pattern"] },
   },
   {
@@ -109,6 +110,7 @@ function normalizeTool(tool) {
     description,
     descLower: description.toLowerCase(),
     parameters: tool.parameters,
+    schemaError: tool.schemaError,
     sourcePath: sourcePathOf(tool),
   };
 }
@@ -157,33 +159,6 @@ export function searchCatalog(catalog, query, limit = 12) {
   return scored.slice(0, Math.max(1, limit)).map(({ score: _s, ...hit }) => hit);
 }
 
-function fieldSummary(key, schema, required) {
-  const s = schema && isObject(schema) ? schema : {};
-  // Only signal-bearing keys: `required:false` and empty descriptions cost tokens and say nothing.
-  return {
-    type: s.type || (Array.isArray(s.anyOf) ? "union" : "unknown"),
-    required: required.has(key) || undefined,
-    description: isString(s.description) ? s.description.slice(0, 120) : undefined,
-  };
-}
-
-function schemaSummary(parameters) {
-  if (!parameters || !isObject(parameters)) return { type: "unknown" };
-  const props = parameters.properties;
-  if (!props || !isObject(props)) {
-    return {
-      type: parameters.type || "object",
-      note: "schema present (no enumerable properties)",
-    };
-  }
-  const required = new Set(Array.isArray(parameters.required) ? parameters.required : []);
-  const fields = {};
-  for (const [key, schema] of Object.entries(props)) {
-    fields[key] = fieldSummary(key, schema, required);
-  }
-  return { type: "object", fields };
-}
-
 /** Optimal string alignment distance: insert/delete/substitute/adjacent-transpose cost 1. */
 function editDistance(a, b) {
   const rows = Array.from({ length: a.length + 1 }, (_, i) => [i, ...new Array(b.length).fill(0)]);
@@ -200,7 +175,7 @@ function editDistance(a, b) {
 }
 
 /** Closest tool names for a mistyped name: substring hits first, then a length-scaled edit distance. */
-export function suggestNames(name, candidates, limit = 3) {
+function suggestNames(name, candidates, limit = 3) {
   const needle = String(name || "").toLowerCase();
   if (!needle) return [];
   const maxDistance = Math.max(1, Math.floor(needle.length / 3));
@@ -231,12 +206,13 @@ export function describeTool(catalog, name) {
   if (!row) {
     return { ok: false, error: unknownToolMessage(name, catalog.map((t) => t.name)) };
   }
+  if (!isObject(row.parameters)) return { ok: false, name, error: "tool schema unavailable: " + (row.schemaError ?? name) };
   if (!row._described) {
     row._described = {
       ok: true,
       name: row.name,
       description: row.description,
-      parameters: schemaSummary(row.parameters),
+      parameters: row.parameters,
       sourcePath: row.sourcePath,
     };
   }