@thurstonsand/pi-librarian 0.1.0

package/AGENTS.md

@@ -0,0 +1,21 @@
+# AGENTS.md
+
+`pi-librarian` gives a pi session the ability to research open source code: deep dives into specific repos ("how does feature X work in repo Y") and discovery across the ecosystem ("compare the most popular SQL ORMs for TypeScript"). It exposes a `librarian` tool that spawns a nested research agent with purpose-built GitHub tools, and a `/librarian` command that attaches those tools directly to the main session.
+
+## Project context
+
+See @CONTEXT.md for project vocabulary.
+
+## Ethos
+
+Agents thrive on reading code for themselves, and projects ALWAYS involve external dependencies. Thus, it's natural to think that it should be easy to point an agent at source code for everything that is involved in a project, even if it's not readily available. Sure, you can always point an agent at a git repo and have it clone it down and start poking around, but that gets cumbersome to type out every time, and most of the time, this exploratory work arrives at a specific answer that doesn't need all of the fidelity of hundreds to thousands of lines of source code to back it up. So it seems natural to give agents a tool that lets them learn about any code external to its own repo that deliver the results without any of the interim.
+
+And this doesn't even touch on the advantages of giving the agent an efficient tool for doing cross-repo comparisons, tracing, market research, etc.
+
+## Core principles
+
+- Build on pi-native concepts, types, and extension APIs where available; read pi source when helpful
+- The librarian is read-only: no write/edit tools, clones live only in the checkout cache
+- Prioritize ergonomics of the exposed interaction surface over internal implementation
+
+See @DEV.md for code style and development commands.

package/CONTEXT.md

@@ -0,0 +1,32 @@
+# Context
+
+## Language
+
+**Librarian**:
+The research agent spawned by the `librarian` tool; runs in a nested, in-memory agent session with its own toolset and system prompt.
+
+**Librarian run**:
+One invocation of the librarian: query in, findings out, with a recorded tool-call trace.
+_Avoid_: session (reserved for pi sessions)
+
+**Run id**:
+The pi session uuid of a librarian run, so that it may be continued for follow up questions.
+
+**Findings**:
+The structured output of a librarian run, produced by `provide_results`: summary, locations, optional description.
+
+**Repo tools**:
+The repository research tools this package registers: `search_repos`, `search_code`, `search_github_code`, `checkout_repo`, `read_github_file`, `provide_results`. Exclusive to librarian runs unless attached.
+
+**Inherited tools**:
+Pi built-ins granted to librarian runs (`read`, `grep`, `find`, `ls`, `bash`) plus tools from allowlisted extensions. Never `write`/`edit`.
+
+**Attach / attached tools**:
+Loading the repo tools into the main pi session via `/librarian`, making them directly usable by the main agent alongside the `librarian` tool.
+_Avoid_: load, enable
+
+## Relationships
+
+- A **Librarian run** ends with exactly one **Findings**, containing zero or more **Locations**.
+- **Attached tools** are the same **Repo tools** a **Librarian** uses, minus `provide_results`.
+- a **Librarian run** is persisted in its own session directory, identified by a **Run id**.

package/DEV.md

@@ -0,0 +1,41 @@
+# DEV.md
+
+## Commands
+
+```bash
+# Full quality gate — run before committing
+npm run check
+
+# Individual steps
+npm run lint
+npm run format
+npm run typecheck
+npm test
+
+# Single test by name pattern
+npm test -- -t "parses sourcegraph stream matches"
+
+# Single test file
+npm test -- test/search-code.test.ts
+```
+
+No build/compile step — the pi framework loads extensions directly from TypeScript source.
+
+## Code Style
+
+- Use TypeBox to ensure runtime type safety
+- Do not change production types to make tests easier; mock the real type instead.
+- Never be afraid to break backwards compatibility if it serves to better solve the current goal
+- Avoid `Pick`, `Omit`, `Partial`, `ReturnType`, indexed-access type derivations like `Foo["bar"]`, other kinds of utility-type derivations unless they are clearly justified.
+- use `.ts` extensions for repo-local imports
+
+## Project structure
+
+- **Entrypoint**: `extensions/librarian.ts` — registers the `librarian` tool and `/librarian` command.
+- **Research tools**: `extensions/librarian/tools/` — `search_repos`, `search_code`, `checkout_repo`, `read_github_file`, `provide_results`.
+- **Clients**: `extensions/librarian/github.ts` (REST + gh auth), `extensions/librarian/sourcegraph.ts` (stream search API).
+- **Checkout cache**: `extensions/librarian/checkout.ts` — blob-less partial clones under `~/.cache/pi-librarian/repos/`.
+- **Runtime**: `extensions/librarian/run.ts` — nested agent session, `provide_results` enforcement.
+- **Presentation**: `extensions/librarian/view.ts` — tool-call trace and findings rendering.
+- **Attach**: `extensions/librarian/attach.ts` — `/librarian` toggle with session-entry persistence.
+- **Settings**: `extensions/librarian/settings.ts` — `librarian.*` keys in pi's global settings.

package/README.md

@@ -0,0 +1,43 @@
+# pi-librarian
+
+A GitHub research subagent for the [pi coding agent](https://github.com/badlogic/pi-mono) inspired by [Amp](https://ampcode.com/): deep-dive questions about specific repos ("how does drizzle-orm implement prepared statements?") and discovery across the ecosystem ("compare the most popular TypeScript SQL ORMs").
+
+## How it works
+
+The `librarian` tool spawns a nested research agent with purpose-built tools:
+
+- **`checkout_repo`** — clone into a local cache; pi's own `grep`/`read`/`find` then work on real files, and `git log -S`/`blame`/`diff` cover history.
+- **`search_repos`** — GitHub repository discovery (stars, topics, language).
+- **`search_code`** — cross-repo public code search via [Grep](https://grep.app/) (regex, global discovery, repo/language/path filters).
+- **`search_github_code`** — GitHub REST code search over public code and private repositories your configured GitHub auth can access.
+- **`read_github_file`** — single-file API reads for quick peeks without cloning.
+
+Private repos work through your existing `gh` auth for GitHub-backed tools.
+
+## Usage
+
+- Ask pi a question involving other repos; it delegates to the `librarian` tool.
+- `/librarian` attaches the research tools directly to your session for manual lookups.
+
+## Configuration
+
+In pi's global `settings.json`:
+
+```jsonc
+{
+  "librarian": {
+    "model": "anthropic/claude-sonnet-5", // default: current session model
+    "thinkingLevel": "high", // default: current session thinking level
+    "extensions": ["~/.pi/agent/extensions/parallel-web-tools"], // extra tools for the librarian
+    "disabledTools": [], // inherited built-ins to drop (write/edit are always excluded)
+    "cacheDir": "/tmp/pi-librarian" // clone cache location
+  }
+}
+```
+
+## Development
+
+```bash
+npm run check                       # biome + tsc + vitest
+pi -e ./extensions/librarian.ts     # run pi with this extension loaded
+```

package/RELEASE.md

@@ -0,0 +1,13 @@
+<!-- markdownlint-disable MD024 -->
+
+# Release notes
+
+## 0.1.0
+
+Initial release of `@thurstonsand/pi-librarian`.
+
+### Added
+
+- Added the `librarian` tool for nested GitHub research runs.
+- Added repo research tools for repository search, code search, checkout, GitHub file reads, and structured findings.
+- Added `/librarian` attached-tool support for using repo tools directly in pi sessions.

package/docs/designs/01-librarian.md

@@ -0,0 +1,215 @@
+# Librarian: GitHub research subagent
+
+## Status
+
+Accepted
+
+## Decision Summary
+
+Build a `librarian` tool: a nested research agent that answers questions about GitHub code — both deep dives into specific repos and discovery across the ecosystem. Reliability comes from inverting the usual approach: instead of leaning on GitHub's search APIs, the librarian clones repos locally and researches them with pi's own battle-tested file tools, reserving search APIs for discovery and candidate-finding. Results are returned through a structured `provide_results` tool, enforced by the runtime.
+
+## Problem Statement / Background
+
+Some of the most useful context for coding work lives in *other people's repos*: how a framework implements a feature, what an error deep in a dependency actually means, which of several libraries fits a need. Two concrete scenarios drove this design:
+
+- **Known-repo deep dive**: "How does drizzle-orm handle prepared statements?" — requires mapping an unfamiliar repo, tracing a flow, and citing exact locations.
+- **Ecosystem discovery**: "Compare the most popular SQL ORMs for TypeScript" — requires popularity signal, docs context, and enough source inspection to compare honestly.
+
+The prior attempt ([default-anton/pi-librarian](https://github.com/default-anton/pi-librarian)) gave a subagent freeform bash and told it to use `gh search code`. That endpoint is GitHub's *legacy* search engine: default-branch only, files under 384 KB, tokenized matching (no regex), skips less-active repos, 10 requests/minute. Roughly half of all runs ended in "I can't find that" — the model was handed a bad instrument. Its presentation layer was also unsatisfying.
+
+Amp's librarian, by contrast, works essentially every time. Investigation (binary string extraction plus interrogating the subagent itself) revealed its shape: seven *structured* per-repo tools — `read_github` (line-ranged reads), `glob_github`, `list_directory_github`, `search_github` (single-repo, paginated), `commit_search`, `diff`, `list_repositories` — plus `web_search`/`read_web_page`, all against GitHub's code-search index, with a subagent prompt oriented around evidence-backed, line-cited findings. The lesson: structure and evidence discipline, not secret infrastructure.
+
+This design adopts Amp's structure where it earns its keep and beats it on the weak link: within-repo search runs on local clones instead of any search API.
+
+## Goals
+
+- Deep-dive questions about a specific repo succeed reliably, with line-cited evidence — the "can't find that" failure mode is eliminated for reachable repos.
+- Discovery questions get real popularity/ecosystem signal (stars, topics, web context), not just code matches.
+- Findings return in a structured shape the main agent and the renderer can both consume.
+- The user can watch research progress live and expand to a full trace.
+- Private repos work through existing `gh` auth with zero extra setup.
+
+## Non-Goals
+
+- Not a general web-research agent; web tools are supporting instruments for GitHub research.
+- No writing, PR creation, or repo mutation — the librarian is read-only.
+- No runtime verification of citations (v1 trusts the prompt; see Alternatives).
+- No turn/cost budget (v1; add if it proves a problem in practice).
+- No replication of Amp's `commit_search`/`diff` as dedicated tools — git in a local clone covers history archaeology.
+
+## Exposed Shape
+
+### Main session surface
+
+- **`librarian` tool** — `{ query, repos?, owners? }`. Spawns a librarian run; returns the structured findings. Description steers the main agent: multi-step GitHub research, unknown locations, cross-repo questions.
+- **`/librarian` command** — attaches the librarian's repo tools to the main session (via `setActiveTools`), so the main agent wields them directly for quick lookups. Run again to detach. Attach state persists across restarts via a custom session entry. The subagent tool remains available while attached; tool descriptions delineate quick lookup vs. delegated research.
+
+### Librarian run toolset
+
+- **Inherited built-ins**: `read`, `grep`, `find`, `ls`, `bash`. Never `write`/`edit`.
+- **Allowlisted extensions** (`librarian.extensions` config): extension paths loaded into the run; defaults to the user's web tools (e.g. parallel-web-tools, whose `fetch_web` is GitHub-aware: issues, PRs, READMEs).
+- **Repo tools** (registered by this package, exclusive to librarian runs unless attached):
+  - `search_repos(query, language?, topic?, sort?, limit?)` — GitHub `/search/repositories`; star-ranked discovery.
+  - `search_code(pattern, regex?, repo?, language?, path?)` — cross-repo public code search through Grep's public MCP endpoint; regex/global discovery plus repo/language/path filters.
+  - `search_github_code(pattern, repos?, owners?, language?, path?, limit?)` — GitHub REST code search over public code and private repositories visible to configured GitHub auth; `repos` entries are `{ owner, repo }`.
+  - `checkout_repo(repo, ref?)` — partial clone into the checkout cache; returns absolute local path, HEAD sha, default branch.
+  - `read_github_file(owner, repo, path, ref?, range?)` — contents-API single-file read without cloning.
+  - `provide_results(summary, locations, description?)` — the mandatory structured finish.
+
+### `provide_results` schema
+
+- `summary` — 1–3 sentence direct answer to the query. No preamble.
+- `locations` — `[{ repo, file, lines?, note }]`; `lines` is a `"start-end"` range enabling GitHub blob links pinned to the checked-out sha.
+- `description` — optional extended findings in markdown (e.g. step-by-step flow tracing).
+
+### Configuration
+
+- `librarian.model` — `provider/model-id` reference; resolved configured → current session model (same machinery as pi-sessions auto-title).
+- `librarian.thinkingLevel` — optional pi thinking level (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`); resolved configured → current session thinking level.
+- `librarian.extensions` — extension paths loaded into librarian runs.
+- `librarian.disabledTools` — inherited built-ins to drop (default: none beyond the hardcoded write/edit exclusion).
+
+### Presentation
+
+Each tool call renders as one line: `verb subject (result summary) timer`, with paths under the checkout cache relativized to `repo/path`:
+
+```
+checkout  drizzle-team/drizzle-orm@main (cached · 2.1k files)       3.2s
+search    code /prepare\w+\(/ in drizzle-team/* (23 hits · 4 repos)  0.9s
+grep      "prepareQuery" drizzle-orm/src (14 matches)                0.3s
+read      drizzle-orm/src/pg-core/session.ts:80-140                  0.1s
+bash      git log -S prepareQuery --oneline (12 commits)              0.7s
+web       "typescript orm comparison 2026" (8 results)                1.1s
+results   3 locations                                                 0.1s
+```
+
+- **Running, collapsed**: query pinned in header; last 3 tool calls with per-call timers; footer `N tool calls · total time · model (thinkingLevel)`.
+- **Expanded** (Ctrl+O): full trace, same header/footer.
+- **Done, collapsed**: header keeps the query; body becomes `findings.summary` only. Expanded results show locations and description. Footer retains call count, total time, and `model (thinkingLevel)`.
+- Failed calls render in red with the error in place of the result summary. Bash commands are whitespace-normalized and truncated (~120 chars).
+
+## Design Decisions
+
+### 1. Clone-first within-repo research
+
+`checkout_repo` performs a `git clone --filter=blob:none` into `/tmp/pi-librarian/repos/<repo-key>` by default, then pi's own `read`/`grep`/`find`/`ls` operate on the clone. `owner/repo` inputs resolve to GitHub; HTTPS and SSH repository URLs are accepted for other Git hosts and are keyed as `<host>/<path>`, allowing nested GitLab-style group paths. GitHub URLs remain constrained to `owner/repo`. No search API sits in the critical path of the primary use case. Blob-less partial clone brings full commit history with lazily-fetched file contents, so `git log -S`, `git blame`, and `git diff` work via bash — covering Amp's `commit_search` and `diff` tools for free, on any ref (Amp is default-branch-only). The cache is refreshed by fetch when stale (15-minute debounce) and has no eviction beyond normal temp-directory cleanup.
+
+**Tradeoff**: disk usage and initial clone latency on monorepo-scale targets, mitigated by blob-less clones and `read_github_file` for single-file questions.
+
+### 2. `search_code` is a candidate-finder, demoted from evidence source
+
+Cross-repo code search is the one job that requires an API, and every available API has gaps. So the prompt frames `search_code` hits as *candidates to verify* via checkout/read — never as citable evidence. This is the root-cause fix for pi-librarian's failure mode: it treated a lossy index as ground truth.
+
+Backend selection is explicit, not hidden fallback. `search_code` uses Grep for public GitHub code search, including regex and broad/no-repo discovery. `search_github_code` uses GitHub REST code search for authenticated/public GitHub searches, especially private repositories or GitHub-specific visibility. The model chooses the instrument based on the task, and failures are reported directly instead of silently trying another backend.
+
+Grep is accessed through the official public MCP endpoint (`https://mcp.grep.app`) as a plain HTTP JSON-RPC backend, not loaded dynamically as an MCP server. GitHub REST code search remains literal/tokenized and does not support regex, but it is the private-repo path through existing `gh`/token auth.
+
+**Risk accepted**: the anonymous endpoint has undocumented rate limits and could be restricted someday; the GitHub fallback keeps the tool functional, and the primary flow (clone-first) doesn't depend on it at all.
+
+### 3. Nested agent session, not tool proxying
+
+A librarian run is a `createAgentSession` with `SessionManager.inMemory()`, an isolated system prompt, and built-in tools selected by name. `SessionManager.inMemory(cacheDir)` uses `cacheDir` as the nested session cwd; it does not persist a session file. Pi offers no API to execute another extension's tool from outside (`getAllTools()` is metadata-only), so "inheriting" main-session extension tools means *loading extension code* into the nested session.
+
+Full inheritance of the user's global extension dir was rejected: it would activate unrelated hooks (UI companions, session recovery, powerline) inside every headless research run, and the librarian would have to exclude itself to avoid recursion. Instead, `librarian.extensions` is an explicit allowlist of paths loaded via `additionalExtensionPaths` with `noExtensions: true`. Predictable composition; the cost is one settings entry when a new tool should flow in.
+
+`excludeTools` and `setActiveToolsByName` serve different layers. `excludeTools` prevents hard-denied tools from being registered into the nested session at all (`write`/`edit`, plus user-disabled tools). After extension loading, the run turns on every remaining registered tool so allowlisted extension tools are usable without needing a second per-tool allowlist.
+
+Pi tool fields split model-facing text: `description` is the tool schema description sent with the provider tool definition, while `promptSnippet` is optional prose for pi's generated "available tools" section. Built-in tools set both; custom tools without `promptSnippet` are still callable but omitted from that prose section.
+
+### 4. Structured finish with enforced `provide_results`
+
+The librarian must end by calling `provide_results`. The tool returns `terminate: true`, which asks pi's agent loop to stop after that tool batch; the runtime still checks after each turn and, if the run went idle without the call, sends a reminder user message — at most 3 times — then returns an error result carrying `session.getLastAssistantText()`. This makes "did the librarian actually answer?" a structural property instead of a parsing hope, and gives the renderer a reliable shape.
+
+Citation honesty ("only cite files you actually read") is enforced in the prompt only. Runtime verification against the tool-call log was considered and deferred (see Alternatives).
+
+### 5. No turn cap
+
+pi-librarian capped runs at 10 turns, which contributed to thin answers. v1 imposes no turn or time budget (KISS); abort remains available via the parent tool-call signal, and a cap is trivially retrofittable if runaway runs materialize.
+
+### 6. Model and thinking resolution mirror auto-title/handoff
+
+`librarian.model` is parsed as a `provider/model-id` `ModelReference` and matched against `ctx.modelRegistry.getAvailable()`; unset or unavailable falls through to the current session's model. `librarian.thinkingLevel` is resolved alongside the model; unset inherits `pi.getThinkingLevel()` and the nested agent receives it via `createAgentSession({ thinkingLevel })`. pi clamps unsupported levels to the selected model's capabilities. Research quality tracks the user's chosen agent quality by default. No hardcoded fallback model list, and none of pi-librarian's 282-line multi-provider failover in v1.
+
+### 7. `/librarian` attach mode
+
+`setActiveTools` verified to work live (rebuilds the system prompt; takes effect next turn) but does not persist. The extension records attach state as a custom session entry and re-attaches on session load — a pattern proven throughout pi-sessions. Attach and subagent coexist: raw tools for single lookups without subagent latency; the `librarian` tool for context-isolated multi-step research.
+
+## Edge Cases & Failure Modes
+
+- **Librarian never calls `provide_results`**: up to 3 reminder messages, then error result containing the raw final text.
+- **Private repo**: `search_github_code`, `checkout_repo`, and `read_github_file` work via existing `gh`/git auth. `search_code` is public-only through Grep. Access failure (403/404) is reported as a finding constraint, not a crash.
+- **Repo doesn't exist / ref invalid**: `checkout_repo` returns a structured error the librarian can react to (e.g. re-resolve via `search_repos`).
+- **GitHub code-search rate limit (10/min)**: tool result carries the retry-after signal; prompt steers toward clone-first anyway.
+- **Grep unavailable/throttled**: `search_code` reports the failure directly; the librarian can choose `search_github_code`, `search_repos`, checkout, or web tools as another avenue.
+- **Monorepo-scale clone**: blob-less clone bounds the damage; single-file questions route to `read_github_file`.
+- **Stale cache**: fetch on checkout when older than 15 minutes; `ref` checkout happens after fetch.
+- **Abort mid-run**: parent tool-call abort signal disposes the nested session; partial trace is preserved in the render with status aborted.
+- **Recursion**: the `librarian` tool is not registered inside librarian runs; the package never loads itself via `librarian.extensions`.
+- **Attach + subagent confusion**: tool descriptions explicitly scope raw tools to single lookups and the subagent to research tasks.
+
+## Alternatives
+
+### Replicate Amp's API-only toolset 1:1
+
+- **Status:** Rejected
+- **Decision or open issue:** Amp's `search_github` rides GitHub's modern code-search index server-side; the public REST equivalent is the legacy engine that caused pi-librarian's failures. Without Amp's infrastructure, the API-only shape inherits the weak link.
+- **Retained discussion:** Amp's *tool structure* (per-repo scoping, pagination, line-ranged reads) is adopted; only the retrieval substrate differs.
+
+### Scrape GitHub's Blackbird web search endpoint
+
+- **Status:** Rejected
+- **Decision or open issue:** The web UI's search (regex-capable) has no official API; scraping it is fragile and ToS-adjacent. Grep's MCP endpoint provides the public regex/global discovery path without scraping GitHub internals.
+
+### Full inheritance of main-session extensions
+
+- **Status:** Rejected
+- **Decision or open issue:** No proxy API exists, so inheritance means loading extension code — and the user's global dir contains UI/session-machinery extensions whose hooks misbehave in headless nested runs. Allowlist chosen; revisit only if the settings-entry friction proves real.
+
+### Runtime citation verification
+
+- **Status:** Deferred
+- **Decision or open issue:** Cross-checking `provide_results.locations` against the run's tool-call log would kill hallucinated citations structurally. Deferred for KISS; the bookkeeping (files read, search hits seen, checkouts performed) is cheap to add later since all events already flow through the run's subscription.
+- **Next step:** Revisit if fabricated paths appear in real usage.
+
+### Turn budget
+
+- **Status:** Deferred
+- **Decision or open issue:** No cap in v1 by explicit choice. Add a config-overridable cap with an end-of-budget reminder if runaway runs occur.
+
+## Implementation Plan
+
+- [x] Phase 1: Repo scaffold
+  - Goal: A checkable, empty pi extension package mirroring pi-sessions conventions.
+  - Files: `package.json` (`@thurstonsand/pi-librarian`, `pi.extensions` entry), `tsconfig.json`, `biome.json`, `AGENTS.md`, `DEV.md`, `.gitignore`, `extensions/librarian.ts` (entrypoint registering nothing yet), `test/` with a placeholder test.
+  - Work: Copy pi-sessions' toolchain shape (biome, tsc, vitest, `npm run check`); pin `@earendil-works/*` peers at the current pi version; TypeBox for all runtime boundaries.
+  - Validation: `npm run check` passes on the empty package; pi loads the extension without error.
+
+- [x] Phase 2: GitHub + Grep clients and the research tools
+  - Goal: `search_repos`, `search_code`, `checkout_repo`, `read_github_file` as pure, individually testable modules with TypeBox param/result schemas.
+  - Files: `extensions/librarian/tools/*.ts`, `extensions/librarian/github.ts`, `extensions/librarian/grep-app.ts`, `extensions/librarian/checkout.ts`, `test/*`.
+  - Work: gh-token-aware GitHub REST client (repo search, contents, explicit code search); Grep MCP-backed public code search; blob-less clone + fetch-debounce cache keyed `owner/repo`; structured errors (not-found, auth, rate-limit) as tool results.
+  - Validation: unit tests with mocked HTTP/git; live smoke: each tool invoked standalone against real repos (public + one private), evidence recorded in `SMOKE.md`.
+
+- [x] Phase 3: Librarian runtime and the `librarian` tool
+  - Goal: End-to-end runs — query in, findings out — with plain-text rendering.
+  - Files: `extensions/librarian/run.ts`, `extensions/librarian/prompt.ts`, `extensions/librarian/model.ts`, `extensions/librarian/results.ts`, `extensions/librarian.ts`.
+  - Work: `createAgentSession` wiring (in-memory session manager, built-ins allowlist minus write/edit, `librarian.extensions` via `additionalExtensionPaths` + `noExtensions`, repo tools via extension factory); system prompt (clone-first strategy, candidate-verification rule, citation discipline, `provide_results` mandate); `provide_results` idle-check with max-3 reminder loop; `librarian.model` resolution; abort propagation and disposal.
+  - Validation: unit tests for the reminder/retry loop and model resolution; live smoke of flows A (drizzle deep dive), B (ORM comparison), C (cross-repo hunt) recorded in `SMOKE.md`.
+
+- [x] Phase 4: Presentation
+  - Goal: The full render spec — collapsed last-3 with timers, expanded trace, findings body, footer.
+  - Files: `extensions/librarian/view.ts`, render wiring in the tool registration.
+  - Work: per-tool-name line formatters (checkout/search/grep/read/read.gh/bash/web/results) with cache-path relativization; timer capture from `tool_execution_start/end`; footer `N tool calls · total time`; done-state body from findings with sha-pinned blob links; red error lines.
+  - Validation: visual smoke in a live pi TUI session, collapsed and expanded, running and done; screenshot or transcript in `SMOKE.md`.
+
+- [x] Phase 5: `/librarian` attach command
+  - Goal: Attach/detach repo tools in the main session, surviving restarts.
+  - Files: `extensions/librarian/attach.ts`, command registration in the entrypoint.
+  - Work: toggle via `getActiveTools`/`setActiveTools` (excluding `provide_results`); persist attach state as a custom session entry; re-attach on session load; adjust tool descriptions to delineate quick lookup vs. delegated research.
+  - Validation: attach, use a raw tool from the main agent, restart pi, confirm tools remain attached; detach and confirm removal.
+
+- [x] Phase 6: Hardening pass against real usage
+  - Goal: The design's edge cases demonstrably handled.
+  - Files: as needed.
+  - Work: exercise private-repo access, rate-limit fallback, invalid repo/ref, abort mid-run, giant-repo checkout; tune the system prompt against observed failure shapes.
+  - Validation: each edge case reproduced and its handling recorded in `SMOKE.md`.

package/docs/designs/02-continuable-runs.md

@@ -0,0 +1,101 @@
+# Continuable librarian runs
+
+## Status
+
+Draft
+
+## Decision Summary
+
+Persist librarian run transcripts as real pi session files in a librarian-owned directory, and add an optional `continue_from: <run uuid>` parameter to the `librarian` tool that reopens a prior run with its full research context. Follow-up questions ("just one more thing about that flow you traced") reuse everything the run already read instead of starting cold. One machinery for both cheap follow-ups and full second research legs.
+
+## Problem Statement / Background
+
+A librarian run often answers 90% of a question, and the main session discovers the missing 10% only after reading the findings — "you traced the D1 prepared-statement flow; does the same apply to batch()?" Today every run is ephemeral (`SessionManager.inMemory`, disposed on completion), so the follow-up pays full price: re-clone context, re-grep, re-read, re-derive. The run's accumulated context — files read, structure understood, dead ends eliminated — is exactly the asset a follow-up needs.
+
+pi already has the machinery: `SessionManager.create(cwd, sessionDir)` persists transcripts to any directory, `SessionManager.open(path)` reopens a specific file, and session files are named `<timestamp>_<sessionId>.jsonl` so a uuid resolves to a file by glob.
+
+## Goals
+
+- A follow-up to a prior run answers from that run's context when sufficient — potentially zero new tool calls — and researches further when not, with no special-casing between the two.
+- The run uuid is discoverable by both the main agent (in the tool result content) and the human (in the rendered output), so continuation survives compaction boundaries: the user can supply the uuid if the agent lost it.
+- Continuations work across pi restarts and from different main sessions.
+
+## Non-Goals
+
+- No integration with pi-sessions indexing/search in v1 (see Alternatives).
+- No transcript eviction policy in v1 (mirrors the checkout-cache decision; `rm -rf /tmp/pi-librarian/sessions` is the manual escape for default settings).
+- No lighter-weight result shape for cheap follow-ups — `provide_results` ceremony applies to every run.
+
+## Exposed Shape
+
+- `librarian` tool gains `continue_from?: string` — the uuid of a prior run. When present, the run resumes that transcript; `query` carries the follow-up question. `repos`/`owners` scope params remain usable.
+- Every run's findings content ends with a `run: <uuid>` line (for the agent), and the done-state render shows the same uuid as a muted line above the footer (for the human).
+- `LibrarianRunDetails` carries `runId`.
+- Tool description documents: pass `continue_from` for follow-ups to earlier research; omit it for new topics.
+
+## Design Decisions
+
+### 1. Transcripts are pi session files in a librarian-owned directory
+
+New runs use `SessionManager.create(cacheDir, <cacheDir>/sessions)` instead of `inMemory`. Continuations glob `<cacheDir>/sessions/*_<uuid>.jsonl` and reopen via `SessionManager.open(path)`. The directory lives outside pi's default session store, so runs never appear in resume pickers and are invisible to pi-sessions indexing.
+
+**Tradeoff**: research memory stays siloed from the pi-sessions "sessions are memory" thesis. Deliberate for v1 — see Alternatives.
+
+### 2. Continuation rebuilds the runtime, the transcript provides the memory
+
+Tools, system prompt, and model are never read from the transcript; each continuation re-registers the current repo tools, rebuilds the system prompt, and resolves the model fresh (configured → current session model — possibly a different model than the original run; pi handles resumed transcripts under a new model). Only the message history is resumed. This keeps continuation semantics identical to fresh runs — including `provide_results` enforcement and the reminder loop — and means prompt/tool improvements apply retroactively to old runs.
+
+### 3. One machinery for cheap follow-ups and big second legs
+
+No fast path. A follow-up that the prior context already answers naturally produces a zero-tool-call run ending in `provide_results`; a big second leg just runs longer. The system prompt gains a continuation expectation: runs may be resumed later; on resume, answer from prior context when it suffices, research further when it does not, and always finish with `provide_results`.
+
+### 4. uuid surfaced in two places, by design redundancy
+
+The agent reads `run: <uuid>` from the result content; the human sees it in the render. If the agent loses the uuid across a compaction boundary, the user can paste it back. Inelegant, deliberately so — one fallback beats zero.
+
+## Edge Cases & Failure Modes
+
+- **Unknown/missing `continue_from` uuid**: structured error result telling the agent the run was not found and to start a fresh run (the sessions dir may have been cleaned).
+- **Concurrent continuations of the same run**: not locked; last write wins. Documented, not defended — a single-user tool.
+- **Original run errored or was aborted**: the transcript still exists and is continuable; the follow-up prompt simply lands after the failure. Often the right move ("try again, but look at X").
+- **Model changed between runs**: allowed; transcript resumes under the newly resolved model.
+- **Aborted continuation**: same abort path as fresh runs; the transcript keeps whatever was persisted, and remains continuable.
+
+## Alternatives
+
+### Persist into the normal pi session store (pi-sessions integration)
+
+- **Status:** Deferred
+- **Decision or open issue:** Landing runs in the regular store would let pi-sessions index them — past research searchable, askable, and linked into lineage. It also pollutes resume pickers with untitled subagent sessions and couples the packages.
+- **Retained discussion:** The cleaner future shape is probably pi-sessions optionally indexing `<cacheDir>/sessions` as an additional source, keeping picker separation while gaining research memory.
+- **Next step:** Revisit once continuations prove their worth in practice.
+
+### In-memory session map for the pi process lifetime
+
+- **Status:** Rejected
+- **Decision or open issue:** Dies on restart and cannot serve a different main session; disk persistence costs nearly nothing given pi's existing machinery.
+
+### Lighter result shape for one-line follow-ups
+
+- **Status:** Rejected
+- **Decision or open issue:** Two result shapes for the main agent to handle; `provide_results` on a zero-tool-call run is cheap enough.
+
+## Implementation Plan
+
+- [ ] Phase 1: Persistent run sessions + `continue_from`
+  - Goal: Runs persist to `<cacheDir>/sessions/`; `continue_from` resumes them end to end.
+  - Files: `extensions/librarian/run.ts`, `extensions/librarian/sessions.ts` (new: dir resolution, uuid→file glob), `extensions/librarian.ts`, `extensions/librarian/prompt.ts`.
+  - Work: swap `SessionManager.inMemory` for `create(cacheDir, sessionsDir)`; capture `session.sessionManager.getSessionId()` — verify the accessor path on the created session — into `LibrarianRunDetails.runId` and append `run: <uuid>` to findings content; add `continue_from` param, resolve uuid→file, `SessionManager.open`, structured not-found error; system prompt continuation expectation; tool description guidance.
+  - Validation: unit tests for uuid→file resolution and not-found handling; `npm run check` green.
+
+- [ ] Phase 2: Render the run id
+  - Goal: Human-visible uuid in the done state.
+  - Files: `extensions/librarian/view.ts`.
+  - Work: muted `run <uuid>` line above the footer in the done render (fresh and continued runs alike).
+  - Validation: view unit test; visual check in TUI.
+
+- [ ] Phase 3: Live smoke
+  - Goal: Continuation proven in a real pi session.
+  - Files: `SMOKE.md`.
+  - Work: run a deep dive; ask a follow-up via `continue_from` in the same session (expect few/zero tool calls); restart pi and continue the same run from a fresh session using the rendered uuid; record evidence.
+  - Validation: SMOKE.md entries with captured output.

package/docs/designs/03-librarian-research-tool-hardening.md

@@ -0,0 +1,141 @@
+# Librarian research-tool hardening
+
+## Status
+
+Deferred
+
+## Decision Summary
+
+Capture three future improvements for librarian runs: stricter reusable checkout cache mechanics, AST-aware local-code tools, and richer result metadata that distinguishes candidates from proof. These are not required for the current implementation, but they are the ideas from other librarian/code-research systems that fit this project without weakening the clone-first, read-only model.
+
+## Problem Statement / Background
+
+The current librarian design already makes the main reliability move: within-repo research happens from local blob-less clones, while search APIs are candidate finders. A survey of adjacent implementations suggested several refinements worth preserving before the details are lost:
+
+- Mitsuhiko's librarian skill treats checkouts as stable cached infrastructure: repo references normalize to a predictable cache path, clones are reused, and refreshes are throttled.
+- Code-research tools such as Octocode and codebase-research-agent emphasize exact regions, line anchors, structured tool contracts, and smaller useful code chunks rather than raw search dumps.
+- This project should borrow those patterns only where they reinforce its own shape. It should not become an API-first search tool, a heavy MCP service, or a semantic index whose snippets are mistaken for evidence.
+
+The useful scenario is a repeated question about the same repo: the librarian should reuse the existing clone, make it match upstream exactly, locate the relevant function/class instead of handing back many grep lines, and report whether each tool result is evidence or only a lead.
+
+## Goals
+
+- Reuse cached clones safely while preserving the librarian's read-only research contract.
+- Improve code citations by returning meaningful syntactic regions when local searches find relevant lines.
+- Make tool results more self-describing, especially around completeness, truncation, errors, and whether output is citable proof.
+
+## Non-Goals
+
+- No dependency on Octocode or any external MCP service.
+- No shift away from clone-first within-repo research.
+- No semantic/vector index as part of this design.
+- No repo-profile cache for now; the value is less clear than the three improvements captured here.
+
+## Exposed Shape
+
+These additions would preserve the current user-facing `librarian` tool and `/librarian` attach command. The exposed changes would be inside the repo-tool surface:
+
+- `checkout_repo` becomes stricter about cache reuse and reports exactly what it did.
+- New AST-aware tools, or AST-aware modes on existing tools, expose code regions with file and line ranges.
+- Existing tools return richer structured details so renderers and prompts can distinguish complete results, partial results, errors, candidates, and proof.
+
+## Design Decisions
+
+### 1. Cached checkouts are disposable infrastructure, not worktrees
+
+A reused checkout should be forced back to the requested upstream state. Unlike a human working clone, librarian checkouts are cache infrastructure. If a previous run or aborted process left local changes, generated files, or an old branch behind, `checkout_repo` should not preserve them.
+
+For default-branch checkouts, the desired shape is:
+
+1. verify the cache path is a git repo for the requested normalized remote;
+2. fetch from origin, with a TTL/debounce to avoid needless network work;
+3. checkout the default branch;
+4. hard reset to `origin/<defaultBranch>`;
+5. remove untracked files;
+6. return the pinned HEAD sha and cache status.
+
+For explicit refs, the tool should fetch the requested ref when needed, force checkout/detach to the resolved commit, and clean the tree. If the cached remote does not match the requested repo, the safe behavior is to discard the cache path and clone again.
+
+**Tradeoff:** this destroys any local modifications inside the checkout cache. That is acceptable because librarian runs are read-only by contract; cache contents are not user-owned work.
+
+### 2. AST-aware tools should return useful code units, not just matches
+
+Plain grep is still the right baseline because it is universal, fast, and transparent. The missing primitive is a way to turn a found line into the smallest useful enclosing code unit: function, method, class, interface, test case, or other language-relevant node.
+
+A future AST-aware addition could be either:
+
+- a new tool such as `read_enclosing_node(path, line, language?)`; or
+- AST-aware options on search tools, such as `context: "enclosing_node"`.
+
+The first version should target high-value languages in this project’s research patterns, likely TypeScript/JavaScript first. `ast-grep` is a plausible implementation substrate because it is language-aware without requiring a bespoke parser per language.
+
+Fallback behavior matters: when parsing fails or the language is unsupported, the tool should return a normal line window with explicit metadata saying it is a fallback. The output must still cite file and line ranges.
+
+**Tradeoff:** AST tools add dependency and language-support complexity. They earn their place only if they reduce token waste and improve citation quality over grep/read chains.
+
+### 3. Tool results should distinguish candidates from proof
+
+The current prompt already says `search_code` results are candidates to verify, not evidence. That discipline should be reflected in tool-result metadata, not only in prose.
+
+Useful metadata fields include:
+
+- `evidenceKind`: `candidate` or `proof`;
+- `complete`: whether the result set or file range is complete;
+- `truncated`: whether output was clipped;
+- `continuation`: how to read more when output is partial;
+- `recoveryHint`: what to try after an empty or failed result;
+- `resolvedRef`/`headSha`: the commit or ref the result is tied to when available.
+
+This would make the renderer clearer and give the prompt a structural hook for citation honesty. Search tools produce candidates. Checked-out file reads and AST-enclosing-node reads produce proof when tied to a concrete path and line range.
+
+**Tradeoff:** richer metadata makes tool details more verbose and requires tests across more branches. The payoff is fewer ambiguous outputs and less prompt-only enforcement.
+
+## Edge Cases & Failure Modes
+
+- **Cached repo remote mismatch:** discard and reclone rather than trying to repair an ambiguous cache entry.
+- **Dirty cached checkout:** hard reset and clean; do not preserve changes.
+- **Fetch fails but stale clone exists:** report a structured error instead of silently researching stale code, unless a future explicit offline mode exists.
+- **Explicit commit not reachable after fetch:** return a ref-resolution error with a recovery hint to verify the ref or repo.
+- **AST parse failure:** fall back to a line window and mark the result as a fallback, not an AST-derived region.
+- **Search result clipped by limits:** mark it as partial and provide the next action instead of implying completeness.
+
+## Alternatives
+
+### Keep checkout cache reuse as-is
+
+- **Status:** Open
+- **Open Issue:** The current implementation already does a force checkout/reset for the default branch and force checkout for explicit refs, but it does not yet fully encode the stricter cache-infrastructure policy described here.
+- **Discussion:** If real-world use never leaves dirty/untracked cache state or remote mismatches, the current behavior may be enough. The stricter policy is still easier to reason about.
+- **Next step:** Audit the current checkout implementation against this design before changing it.
+
+### Add Octocode as a dependency
+
+- **Status:** Rejected
+- **Decision**: Octocode is an external code-research/MCP-style system, not a pi-native tool. Its useful contribution here is contract design: exact regions, line anchors, pagination, bulk/error semantics, and candidate-vs-proof discipline.
+- **Discussion:** Borrow the result-shape ideas, not the infrastructure.
+
+### Add repo profiles
+
+- **Status:** Rejected
+- **Decision**: Repo profiles are less compelling for now than cache correctness, AST-aware reads, and result metadata.
+- **Discussion:** A profile cache could store package manager, source roots, manifests, and test commands, but it risks becoming stale orientation data that agents over-trust. Evidence should continue to come from actual file reads.
+
+## Implementation Plan
+
+- [ ] Phase 1: Checkout cache hardening
+  - Goal: Make reused clones deterministic infrastructure that always match the requested upstream ref.
+  - Files: `extensions/librarian/checkout.ts`, `extensions/librarian/tools/checkout-repo.ts`, checkout tests.
+  - Work: Verify cached remote identity; discard/reclone on mismatch; fetch with existing debounce; hard reset and clean for default branch and explicit refs; include cache action metadata in tool details.
+  - Validation: Unit tests for dirty checkout cleanup, untracked file cleanup, remote mismatch reclone, explicit ref checkout, and fetch failure behavior; `npm run check`.
+
+- [ ] Phase 2: Result metadata contract
+  - Goal: Encode candidate/proof and completeness semantics in tool details.
+  - Files: repo tool modules under `extensions/librarian/tools/`, `extensions/librarian/view.ts`, prompt/tests.
+  - Work: Add shared metadata fields where useful; mark `search_code` as candidate; mark checked-out/read file outputs as proof when they include concrete refs and line ranges; expose truncation and continuation hints.
+  - Validation: Tool unit tests and view tests showing partial/candidate/error states; `npm run check`.
+
+- [ ] Phase 3: AST-aware local-code primitive
+  - Goal: Provide a first syntactic-region helper for cloned repos.
+  - Files: new tool module under `extensions/librarian/tools/`, tool registration, prompt/tests.
+  - Work: Choose implementation substrate, likely `ast-grep`; support TypeScript/JavaScript first; return enclosing node line ranges; fall back to a line window with explicit fallback metadata.
+  - Validation: Fixture tests for functions/classes/methods/tests, unsupported language fallback, parse failure fallback, and live smoke on a real TypeScript repo.