@thurstonsand/pi-librarian 0.2.0 → 0.3.0

package/README.md

@@ -2,6 +2,10 @@
 
 A GitHub research subagent for the [pi coding agent](https://github.com/badlogic/pi-mono) inspired by [Amp](https://ampcode.com/): deep-dive questions about specific repos ("how does drizzle-orm implement prepared statements?") and discovery across the ecosystem ("compare the most popular TypeScript SQL ORMs").
 
+![calling the librarian tool](images/librarian-run.png)
+
+![librarian tool results](images/librarian-results.png)
+
 ## How it works
 
 The `librarian` tool spawns a nested research agent with purpose-built tools:
@@ -39,17 +43,19 @@ In pi's global `settings.json`:
     "tools": ["search_web", "fetch_web"],
     "extensions": ["~/.pi/agent/extensions/parallel-web-tools"],
     "cacheDir": "/tmp/pi-librarian",
+    "debug": { "persistRuns": false },
   },
 }
 ```
 
-| Setting         | Recommended                                   | Default                        |
-| --------------- | --------------------------------------------- | ------------------------------ |
-| `model`         | `openai-codex/gpt-5.5`                        | current session model          |
-| `thinkingLevel` | `off`                                         | current session thinking level |
-| `tools`         | names of extra tools to activate, when needed | `[]`                           |
-| `extensions`    | escape hatch paths for tools not loaded in pi | `[]`                           |
-| `cacheDir`      | `/tmp/pi-librarian`                           | `/tmp/pi-librarian`            |
+| Setting             | Recommended                                     | Default                        |
+| ------------------- | ----------------------------------------------- | ------------------------------ |
+| `model`             | `openai-codex/gpt-5.5`                          | current session model          |
+| `thinkingLevel`     | `off`                                           | current session thinking level |
+| `tools`             | names of extra tools to activate, when needed   | `[]`                           |
+| `extensions`        | escape hatch paths for tools not loaded in pi   | `[]`                           |
+| `cacheDir`          | `/tmp/pi-librarian`                             | `/tmp/pi-librarian`            |
+| `debug.persistRuns` | persist nested session file paths for debugging | `false`                        |
 
 `librarian.tools` is the activation gate for extra tools. `librarian.extensions` only adds extension paths to the search space when a named tool is not already loaded in the main pi session; listing an extension path does not activate every tool in that bundle. Librarian runs exclude `write` and `edit`.
 

package/RELEASE.md

@@ -2,6 +2,28 @@
 
 # Release notes
 
+## 0.3.0
+
+Hardens librarian research tools and aligns GitHub file reads with pi's native `read` tool.
+
+### Changed
+
+- Changed `read_github_file` to use `offset` and `limit` instead of `range`, matching pi's native `read` tool semantics.
+- Hardened `checkout_repo` cache reuse so mismatched or malformed cached paths are discarded, while reused checkouts are reset and cleaned before research.
+
+## 0.2.1
+
+Fixes TUI rendering corruption.
+
+### Added
+
+- Added `librarian.debug.persistRuns` to keep nested run session files around for debugging.
+
+### Fixed
+
+- Fixed librarian trace rendering leaving stale "N earlier calls" duplicates in the TUI, caused by unsilenced `@octokit/request` deprecation warnings writing to the console.
+- Fixed `search_code` retries riding out a fixed 30s-per-attempt budget; retries now use an escalating timeout schedule so a stalled request recovers in seconds instead of up to a minute.
+
 ## 0.2.0
 
 Adds continuable librarian runs and opt-in for extra tools.

package/docs/designs/03-librarian-research-tool-hardening.md

@@ -0,0 +1,136 @@
+# Librarian research-tool hardening
+
+## Status
+
+Partially implemented
+
+## Decision Summary
+
+Capture the hardening work implemented for librarian runs: disposable checkout-cache mechanics and remote file reads that behave like pi's native `read` tool. AST-aware local-code tools and heavier result metadata remain deferred ideas, not active implementation pressure.
+
+## Problem Statement / Background
+
+The current librarian design already makes the main reliability move: within-repo research happens from local blob-less clones, while search APIs are candidate finders. A survey of adjacent implementations suggested several refinements worth preserving before the details are lost:
+
+- Mitsuhiko's librarian skill treats checkouts as stable cached infrastructure: repo references normalize to a predictable cache path, clones are reused, and refreshes are throttled.
+- Code-research tools such as Octocode and codebase-research-agent emphasize exact regions, line anchors, and smaller useful code chunks rather than raw search dumps.
+- This project should borrow those patterns only where they simplify the librarian's operating surface. It should not become an API-first search tool, a heavy MCP service, or a semantic index whose snippets are mistaken for evidence.
+
+The useful scenario is a repeated question about the same repo: the librarian should reuse the existing clone, force it back to the requested upstream state subject to fetch debounce, clean away any cache-local debris, and let the agent read GitHub files with the same mental model it uses for local files.
+
+## Goals
+
+- Reuse cached clones safely while preserving the librarian's read-only research contract.
+- Treat checkout-cache contents as disposable infrastructure, not user-owned worktrees.
+- Make `read_github_file` operationally match pi's native `read` tool where possible.
+
+## Non-Goals
+
+- No dependency on Octocode or any external MCP service.
+- No shift away from clone-first within-repo research.
+- No semantic/vector index as part of this design.
+- No AST-aware local-code primitive for now; useful, but not part of this hardening pass.
+- No broad tool-result metadata expansion unless a concrete enforcement feature needs it.
+- No repo-profile cache for now; the value is less clear than the hardening work captured here.
+
+## Exposed Shape
+
+These additions would preserve the current user-facing `librarian` tool and `/librarian` attach command. The exposed changes would be inside the repo-tool surface:
+
+- `checkout_repo` becomes stricter about cache reuse by resetting and cleaning cache entries deterministically.
+- `read_github_file` uses `offset`/`limit` semantics and output conventions that closely mirror pi's native `read` tool.
+- Existing `provide_results.locations` remains the final citation surface. Search results are still candidates by prompt discipline, not by a new metadata framework.
+
+## Design Decisions
+
+### 1. Cached checkouts are disposable infrastructure, not worktrees
+
+A reused checkout should be forced back to the requested upstream state, subject only to the fetch debounce. Unlike a human working clone, librarian checkouts are cache infrastructure. If a previous run, aborted process, or manual edit left local changes, generated files, or an old branch behind, `checkout_repo` should not preserve them. Manual edits inside the checkout cache are undefined behavior.
+
+For default-branch checkouts, the desired shape is:
+
+1. verify the cache path is a git repo for the requested normalized remote;
+2. fetch from origin unless the fetch debounce says the clone is fresh enough;
+3. checkout the default branch;
+4. hard reset to `origin/<defaultBranch>`;
+5. remove untracked files with `git clean -fdx`;
+6. return the pinned HEAD sha and cache status.
+
+For explicit refs, the tool should fetch the requested ref when needed, force checkout/detach to the resolved commit, hard reset to that commit, and clean the tree. If the cached remote does not match the requested repo, or if the cache path is malformed, the safe behavior is to discard the cache path and clone again.
+
+**Tradeoff:** this destroys any local modifications inside the checkout cache. That is acceptable because librarian runs are read-only by contract; cache contents are not user-owned work.
+
+### 2. Remote file reads should behave like local reads
+
+`read_github_file` is most useful when it feels like pi's native `read` tool pointed at GitHub. The agent should not need one mental model for local checkout files and another for remote GitHub files.
+
+The tool should use `offset` and `limit` parameters rather than a bespoke `[startLine, endLine]` range:
+
+- `offset` is a 1-indexed starting line;
+- `limit` is the maximum number of lines to return;
+- default behavior should match native `read` as closely as practical;
+- clipped output should tell the agent how to continue with `offset`/`limit`;
+- file output should be directly returned as text, not written to disk;
+
+Directory listings may remain GitHub-specific, but file reads should be operationally interchangeable with local `read` output. This is more useful than extra metadata because it changes the agent's actual operating surface.
+
+**Tradeoff:** changing from `range` to `offset`/`limit` breaks the current tool schema. That is acceptable if it makes the tool more pi-native and easier for agents to use consistently.
+
+### 3. AST and metadata expansion remain deferred
+
+Plain grep plus `read` remains the right baseline because it is universal, fast, and transparent. A future AST-aware addition could still be useful, probably as `read_enclosing_node(path, line, language?)` for TypeScript/JavaScript first, but it should wait until the simpler read and checkout semantics are solid.
+
+Likewise, broad result metadata such as `evidenceKind`, `complete`, `truncated`, `continuation`, and `recoveryHint` should not be added speculatively. Agents primarily use tool text and prompt instructions; hidden metadata only matters when a renderer or validator consumes it. If citation honesty needs more structure later, the better first step is validating `provide_results.locations` against the run trace: files checked out, files read, line ranges observed, and search hits that were only candidates.
+
+**Tradeoff:** this leaves candidate-vs-proof enforcement mostly in the prompt for now. That is acceptable under KISS until there is a concrete validation feature to support.
+
+## Edge Cases & Failure Modes
+
+- **Cached repo remote mismatch:** discard and reclone rather than trying to repair an ambiguous cache entry.
+- **Dirty cached checkout:** hard reset and `git clean -fdx`; do not preserve changes.
+- **Fetch fails but stale clone exists:** report a structured error instead of silently researching stale code, unless a future explicit offline mode exists.
+- **Explicit commit not reachable after fetch:** return a ref-resolution error with a recovery hint to verify the ref or repo.
+- **Large GitHub file:** return a clipped line window using native-read-like semantics and tell the agent the next `offset`/`limit` to request.
+- **Search result clipped by limits:** keep the text honest about limits; do not add metadata unless a concrete consumer needs it.
+
+## Alternatives
+
+### Keep checkout cache reuse as-is
+
+- **Status:** Rejected
+- **Decision:** The current implementation already does a force checkout/reset for the default branch and force checkout for explicit refs, but it does not fully encode the disposable-cache policy described here.
+- **Discussion:** The stricter policy is easier to reason about and matches the contract: checkout-cache contents are not user-owned work.
+
+### Add Octocode as a dependency
+
+- **Status:** Rejected
+- **Decision**: Octocode is an external code-research/MCP-style system, not a pi-native tool. Its useful contribution here is contract design: exact regions, line anchors, pagination, bulk/error semantics, and candidate-vs-proof discipline.
+- **Discussion:** Borrow only the interaction ideas that make pi-native tools easier to use. Do not import the infrastructure or add metadata for its own sake.
+
+### Add repo profiles
+
+- **Status:** Rejected
+- **Decision**: Repo profiles are less compelling for now than cache correctness and native-read-like remote file access.
+- **Discussion:** A profile cache could store package manager, source roots, manifests, and test commands, but it risks becoming stale orientation data that agents over-trust. Evidence should continue to come from actual file reads.
+
+## Implementation Plan
+
+- [x] Phase 1: Checkout cache hardening
+  - Goal: Make reused clones deterministic disposable infrastructure that match the requested upstream ref, subject to fetch debounce.
+  - Files: `extensions/librarian/checkout.ts`, `extensions/librarian/tools/checkout-repo.ts`, checkout tests.
+  - Work: Verify cached remote identity; discard/reclone on mismatch or malformed cache; fetch with existing debounce; hard reset and `git clean -fdx` for default branch and explicit refs.
+  - Validation: Unit tests for dirty checkout cleanup, untracked file cleanup, remote mismatch reclone, explicit ref checkout, malformed cache replacement, and fetch failure behavior; `npm run check`.
+
+- [x] Phase 2: Native-read-like `read_github_file`
+  - Goal: Make remote GitHub file reads operationally match pi's native `read` tool.
+  - Files: `extensions/librarian/tools/read-github-file.ts`, prompt/tests.
+  - Work: Replace `range` with `offset`/`limit`; match native read truncation and continuation language as closely as practical; keep direct text output and directory listing behavior.
+  - Validation: Tool tests for default truncation, offset/limit reads, clipped continuation text, invalid ranges, directory listings, and pinned `ref` reads; `npm run check`.
+
+- [ ] Deferred: citation validation
+  - Goal: If citation honesty needs structural enforcement, validate `provide_results.locations` against the run trace instead of adding broad metadata first.
+  - Work: Track which checked-out or GitHub files were actually read, including covered line ranges; reject or warn on locations sourced only from search candidates.
+
+- [ ] Deferred: AST-aware local-code primitive
+  - Goal: Provide a syntactic-region helper only if grep/read chains prove too wasteful.
+  - Work: Consider `read_enclosing_node(path, line, language?)`, likely TypeScript/JavaScript first.

package/extensions/librarian/checkout.ts

@@ -148,6 +148,17 @@ async function pathExists(target: string): Promise<boolean> {
   }
 }
 
+async function currentRemoteKey(
+  dest: string,
+  signal: AbortSignal | undefined,
+): Promise<string | undefined> {
+  try {
+    return parseRepoReference(await git(["remote", "get-url", "origin"], dest, signal)).key;
+  } catch {
+    return undefined;
+  }
+}
+
 async function clone(
   reference: RepoReference,
   dest: string,
@@ -207,7 +218,13 @@ export async function checkoutRepo(
   const repo = parseRepoReference(repoInput);
   const dest = repoCachePath(cacheDir, repo.key);
 
-  const reusedClone = await pathExists(path.join(dest, ".git"));
+  const reusedClone =
+    (await pathExists(path.join(dest, ".git"))) &&
+    (await currentRemoteKey(dest, signal)) === repo.key;
+  if (!reusedClone) {
+    await fs.rm(dest, { recursive: true, force: true });
+  }
+
   if (reusedClone) {
     await fetchIfStale(dest, signal);
   } else {
@@ -223,6 +240,7 @@ export async function checkoutRepo(
       // Refs already fetched (e.g. a sha reachable from an earlier fetch) check out directly.
       await git(["checkout", "--force", "--detach", ref], dest, signal);
     }
+    await git(["reset", "--hard", "HEAD"], dest, signal);
     checkedOutRef = ref;
   } else {
     const branch = await defaultBranch(dest, signal);
@@ -231,6 +249,8 @@ export async function checkoutRepo(
     checkedOutRef = branch;
   }
 
+  await git(["clean", "-fdx"], dest, signal);
+
   const headSha = await git(["rev-parse", "HEAD"], dest, signal);
   const files = await git(["ls-files"], dest, signal);
   const fileCount = files.length === 0 ? 0 : files.split("\n").length;

package/extensions/librarian/github.ts

@@ -28,7 +28,14 @@ function repoFullName(repo: GitHubRepoSlug): string {
   return `${repo.owner}/${repo.repo}`;
 }
 
-export type GitHubClientProvider = () => Promise<GitHubClient>;
+export interface GitHubClientApi {
+  searchRepositories(params: SearchRepositoriesParams): Promise<RepoSearchResult>;
+  searchCode(params: SearchGitHubCodeParams): Promise<GitHubCodeSearchResult>;
+  readContents(params: ReadContentsParams): Promise<FileContents | DirectoryContents>;
+  getRepo(repo: GitHubRepoSlug): Promise<RepoMeta>;
+}
+
+export type GitHubClientProvider = () => Promise<GitHubClientApi>;
 
 export async function resolveGitHubToken(): Promise<string | undefined> {
   const envToken = process.env.GITHUB_TOKEN?.trim() || process.env.GH_TOKEN?.trim();
@@ -45,18 +52,26 @@ export async function resolveGitHubToken(): Promise<string | undefined> {
   }
 }
 
+// Any console output corrupts pi's interactive TUI (its differential renderer
+// cannot recover from writes it did not make), so octokit must never log.
+// The top-level `log` alone is not enough: @octokit/request resolves its logger
+// from the per-request options (`request.log || console`) when emitting API
+// deprecation warnings, so the silent logger has to be passed at both levels.
+const SILENT_OCTOKIT_LOG = {
+  debug: () => {},
+  info: () => {},
+  warn: () => {},
+  error: () => {},
+};
+
 export function createGitHubClient(token: string | undefined): GitHubClient {
   return new GitHubClient(
     new Octokit({
       ...(token ? { auth: token } : {}),
       userAgent: "pi-librarian",
-      log: {
-        debug: () => {},
-        info: () => {},
-        warn: () => {},
-        error: () => {},
-      },
+      log: SILENT_OCTOKIT_LOG,
       request: {
+        log: SILENT_OCTOKIT_LOG,
         headers: {
           "x-github-api-version": "2022-11-28",
         },
@@ -241,7 +256,7 @@ function buildGitHubCodeQuery(params: SearchGitHubCodeParams): string {
   return parts.join(" ");
 }
 
-export class GitHubClient {
+export class GitHubClient implements GitHubClientApi {
   constructor(private readonly octokit: Octokit) {}
 
   async searchRepositories(params: SearchRepositoriesParams): Promise<RepoSearchResult> {

package/extensions/librarian/grep-app.ts

@@ -2,7 +2,10 @@ import { Type } from "typebox";
 import { safeParseTypeBoxValue } from "../shared/typebox.ts";
 
 const GREP_MCP_URL = "https://mcp.grep.app";
-const GREP_SEARCH_TIMEOUT_MS = 30_000;
+// Escalating per-attempt budgets: the upstream gateway kills stalled requests at ~15s, so
+// early attempts abort fast (p90 success is ~500ms) instead of waiting out the full timeout.
+const GREP_ATTEMPT_TIMEOUTS_MS = [3_000, 5_000, 7_000];
+const GREP_RETRY_DELAY_MS = 300;
 
 export class GrepError extends Error {
   constructor(message: string) {
@@ -159,7 +162,12 @@ export function parseGrepMcpEvents(body: string): GrepCodeSearchResult {
       continue;
     }
     if (result.isError) {
-      throw new GrepError("Grep MCP returned an error result.");
+      const message = (result.content ?? [])
+        .map((rawContent) => safeParseTypeBoxValue(MCP_TEXT_CONTENT_SCHEMA, rawContent)?.text)
+        .filter((text): text is string => text !== undefined)
+        .join("\n")
+        .trim();
+      throw new GrepError(message || "Grep MCP returned an error result.");
     }
 
     for (const rawContent of result.content ?? []) {
@@ -203,11 +211,45 @@ function buildGrepArguments(params: GrepCodeSearchParams): Record<string, unknow
   return args;
 }
 
-export async function searchCodeGrep(
+function extractHtmlTitle(body: string): string | undefined {
+  const match = /<title[^>]*>(.*?)<\/title>/is.exec(body);
+  return match?.[1]?.replace(/\s+/g, " ").trim();
+}
+
+function formatHttpError(status: number, body: string): string {
+  return `Grep MCP returned ${status}: ${extractHtmlTitle(body) ?? body.slice(0, 200)}`;
+}
+
+function formatTransientError(error: unknown): string {
+  const cause =
+    error instanceof GrepTransientHttpError ? `returned ${error.status}` : "request failed";
+  return `Grep MCP ${cause} after retry (transient backend error). Retry, or simplify the query.`;
+}
+
+function delay(milliseconds: number, signal: AbortSignal | undefined): Promise<void> {
+  if (signal?.aborted) {
+    return Promise.reject(signal.reason);
+  }
+
+  return new Promise((resolve, reject) => {
+    const timeout = setTimeout(resolve, milliseconds);
+    signal?.addEventListener(
+      "abort",
+      () => {
+        clearTimeout(timeout);
+        reject(signal.reason);
+      },
+      { once: true },
+    );
+  });
+}
+
+async function searchCodeGrepAttempt(
   params: GrepCodeSearchParams,
   signal: AbortSignal | undefined,
+  timeoutMs: number,
 ): Promise<GrepCodeSearchResult> {
-  const timeoutSignal = AbortSignal.timeout(GREP_SEARCH_TIMEOUT_MS);
+  const timeoutSignal = AbortSignal.timeout(timeoutMs);
   const requestSignal = signal ? AbortSignal.any([signal, timeoutSignal]) : timeoutSignal;
   const response = await fetch(GREP_MCP_URL, {
     method: "POST",
@@ -230,8 +272,49 @@ export async function searchCodeGrep(
 
   const body = await response.text();
   if (!response.ok) {
-    throw new GrepError(`Grep MCP returned ${response.status}: ${body.slice(0, 200)}`);
+    if (response.status >= 500) {
+      throw new GrepTransientHttpError(response.status, body);
+    }
+    throw new GrepError(formatHttpError(response.status, body));
   }
 
   return parseGrepMcpEvents(body);
 }
+
+// Do not extend GrepError: GrepError is the deterministic no-retry signal.
+class GrepTransientHttpError extends Error {
+  readonly status: number;
+
+  constructor(status: number, body: string) {
+    super(formatHttpError(status, body));
+    this.name = "GrepTransientHttpError";
+    this.status = status;
+  }
+}
+
+export async function searchCodeGrep(
+  params: GrepCodeSearchParams,
+  signal: AbortSignal | undefined,
+): Promise<GrepCodeSearchResult> {
+  if (signal?.aborted) {
+    throw signal.reason;
+  }
+
+  let transientError: unknown;
+  for (const [attempt, timeoutMs] of GREP_ATTEMPT_TIMEOUTS_MS.entries()) {
+    if (attempt > 0) {
+      await delay(GREP_RETRY_DELAY_MS, signal);
+    }
+
+    try {
+      return await searchCodeGrepAttempt(params, signal, timeoutMs);
+    } catch (error) {
+      if (error instanceof GrepError || signal?.aborted) {
+        throw error;
+      }
+      transientError = error;
+    }
+  }
+
+  throw new GrepError(formatTransientError(transientError));
+}

package/extensions/librarian/run.ts

@@ -52,6 +52,7 @@ export interface LibrarianRunDetails {
   checkouts: Record<string, string>;
   error?: string;
   runId?: string;
+  debugSessionPath?: string;
   startedAt: number;
   endedAt?: number;
 }
@@ -108,7 +109,11 @@ export async function runLibrarian(
   };
 
   const finalizeDetails = (status: LibrarianRunStatus, content: string): string => {
-    const resultContent = details.runId ? `${content}\n\nrun: ${details.runId}` : content;
+    const suffixes = [
+      ...(details.runId ? [`run: ${details.runId}`] : []),
+      ...(details.debugSessionPath ? [`debug_session: ${details.debugSessionPath}`] : []),
+    ];
+    const resultContent = suffixes.length > 0 ? `${content}\n\n${suffixes.join("\n")}` : content;
     details.status = status;
     details.endedAt = Date.now();
     if (status !== "done") {
@@ -137,6 +142,10 @@ export async function runLibrarian(
     ? await openContinuedSession(options.continueFrom, sessionsDir, options.settings.cacheDir, fail)
     : SessionManager.create(options.settings.cacheDir, sessionsDir);
   details.runId = sessionManager.getSessionId();
+  const debugSessionPath = sessionManager.getSessionFile();
+  if (options.settings.debug.persistRuns && debugSessionPath) {
+    details.debugSessionPath = debugSessionPath;
+  }
 
   let findings: Findings | undefined;
   const repoTools = [

package/extensions/librarian/settings.ts

@@ -20,6 +20,11 @@ const LIBRARIAN_FILE_SETTINGS_SCHEMA = Type.Object({
   extensions: Type.Optional(Type.Array(Type.String())),
   tools: Type.Optional(Type.Array(Type.String())),
   cacheDir: Type.Optional(Type.String()),
+  debug: Type.Optional(
+    Type.Object({
+      persistRuns: Type.Optional(Type.Boolean()),
+    }),
+  ),
 });
 
 const ROOT_SETTINGS_SCHEMA = Type.Object({
@@ -45,6 +50,9 @@ export interface LibrarianSettings {
   extensions: string[];
   tools: string[];
   cacheDir: string;
+  debug: {
+    persistRuns: boolean;
+  };
 }
 
 export function getDefaultCacheDir(): string {
@@ -117,6 +125,9 @@ export function resolveLibrarianSettings(fileSettings: LibrarianFileSettings): L
     extensions: normalizeExtensionPaths(fileSettings.extensions),
     tools: normalizeToolNames(fileSettings.tools),
     cacheDir: normalizeCacheDir(fileSettings.cacheDir),
+    debug: {
+      persistRuns: fileSettings.debug?.persistRuns ?? false,
+    },
   };
 }
 

package/extensions/librarian/tools/read-github-file.ts

@@ -1,10 +1,13 @@
-import { defineTool } from "@earendil-works/pi-coding-agent";
+import {
+  DEFAULT_MAX_BYTES,
+  defineTool,
+  formatSize,
+  truncateHead,
+} from "@earendil-works/pi-coding-agent";
 import { Type } from "typebox";
 import { GitHubApiError, type GitHubClientProvider } from "../github.ts";
 import { LIBRARIAN_TOOL_NAMES } from "./names.ts";
 
-const MAX_LINES_WITHOUT_RANGE = 1200;
-
 export interface ReadGitHubFileDetails {
   kind: "read_github_file";
   owner: string;
@@ -17,15 +20,24 @@ export interface ReadGitHubFileDetails {
 const ReadGitHubFileParams = Type.Object({
   owner: Type.String({ description: "Repository owner or organization." }),
   repo: Type.String({ description: "Repository name." }),
-  path: Type.String({ description: "File or directory path within the repository." }),
+  path: Type.String({
+    description: "File or directory path within the repository.",
+  }),
   ref: Type.Optional(
-    Type.String({ description: "Branch, tag, or commit sha. Default: the default branch." }),
+    Type.String({
+      description: "Branch, tag, or commit sha. Default: the default branch.",
+    }),
+  ),
+  offset: Type.Optional(
+    Type.Number({
+      minimum: 1,
+      description: "Line number to start reading from (1-indexed).",
+    }),
   ),
-  range: Type.Optional(
-    Type.Array(Type.Number({ minimum: 1 }), {
-      description: "[startLine, endLine] (1-based, inclusive) for large files.",
-      minItems: 2,
-      maxItems: 2,
+  limit: Type.Optional(
+    Type.Number({
+      minimum: 1,
+      description: "Maximum number of lines to read.",
     }),
   ),
 });
@@ -34,19 +46,22 @@ export function createReadGitHubFileTool(githubClient: GitHubClientProvider) {
   return defineTool<typeof ReadGitHubFileParams, ReadGitHubFileDetails>({
     name: LIBRARIAN_TOOL_NAMES.readGitHubFile,
     label: "Read GitHub file",
-    description:
-      "Read a single file (or list a directory) from a GitHub repo via the API, without cloning.",
+    description: `Read a single file (or list a directory) from a GitHub repo via the API, without cloning.`,
     promptSnippet: "Read Github file",
     promptGuidelines: [
       "For quick peeks — package.json, a README, one source file.",
       "For multi-file exploration prefer checkout_repo.",
+      "Use offset/limit for larger files.",
     ],
     parameters: ReadGitHubFileParams,
 
     async execute(_toolCallId, params) {
-      const range = params.range ? (params.range as [number, number]) : undefined;
-      if (range && range[1] < range[0]) {
-        throw new Error("range end must be greater than or equal to range start.");
+      const offset = params.offset ?? 1;
+      if (offset < 1) {
+        throw new Error("offset must be greater than or equal to 1.");
+      }
+      if (params.limit !== undefined && params.limit < 1) {
+        throw new Error("limit must be greater than or equal to 1.");
       }
 
       try {
@@ -75,32 +90,52 @@ export function createReadGitHubFileTool(githubClient: GitHubClientProvider) {
         }
 
         const allLines = contents.text.split("\n");
-        let start = 1;
-        let end = allLines.length;
-        let clipNote = "";
-
-        if (range) {
-          start = Math.min(range[0], allLines.length);
-          end = Math.min(range[1], allLines.length);
-        } else if (allLines.length > MAX_LINES_WITHOUT_RANGE) {
-          end = MAX_LINES_WITHOUT_RANGE;
-          clipNote = `\n... clipped at line ${MAX_LINES_WITHOUT_RANGE} of ${allLines.length}; pass range to read further.`;
+        const startLine = offset - 1;
+        if (startLine >= allLines.length) {
+          throw new Error(
+            `Offset ${offset} is beyond end of file (${allLines.length} lines total)`,
+          );
         }
 
-        const width = String(end).length;
-        const numbered = allLines
-          .slice(start - 1, end)
-          .map((line, index) => `${String(start + index).padStart(width)}\t${line}`)
-          .join("\n");
+        const selectedLines =
+          params.limit === undefined
+            ? allLines.slice(startLine)
+            : allLines.slice(startLine, Math.min(startLine + params.limit, allLines.length));
+        const selectedContent = selectedLines.join("\n");
+        const truncated = truncateHead(selectedContent);
+        let text: string;
+        let lineCount = truncated.outputLines;
+
+        if (truncated.firstLineExceedsLimit) {
+          const firstLineSize = formatSize(Buffer.byteLength(allLines[startLine] ?? "", "utf-8"));
+          text = `[Line ${offset} is ${firstLineSize}, exceeds ${formatSize(DEFAULT_MAX_BYTES)} limit.]`;
+        } else if (truncated.truncated) {
+          const endLine = offset + truncated.outputLines - 1;
+          const nextOffset = endLine + 1;
+          const limitNote =
+            truncated.truncatedBy === "lines" ? "" : ` (${formatSize(DEFAULT_MAX_BYTES)} limit)`;
+          text = `${truncated.content}\n\n[Showing lines ${offset}-${endLine} of ${allLines.length}${limitNote}. Use offset=${nextOffset} to continue.]`;
+        } else if (
+          params.limit !== undefined &&
+          startLine + selectedLines.length < allLines.length
+        ) {
+          const remaining = allLines.length - (startLine + selectedLines.length);
+          const nextOffset = startLine + selectedLines.length + 1;
+          text = `${truncated.content}\n\n[${remaining} more lines in file. Use offset=${nextOffset} to continue.]`;
+          lineCount = selectedLines.length;
+        } else {
+          text = truncated.content;
+          lineCount = selectedLines.length;
+        }
 
         return {
-          content: [{ type: "text", text: `${numbered}${clipNote}` }],
+          content: [{ type: "text", text }],
           details: {
             kind: "read_github_file",
             owner: params.owner,
             repo: params.repo,
             path: params.path,
-            lineCount: end - start + 1,
+            lineCount,
             isDirectory: false,
           },
         };

package/extensions/librarian/tools/search-code.ts

@@ -14,12 +14,11 @@ export interface SearchCodeDetails {
 const SearchCodeParams = Type.Object({
   pattern: Type.String({
     description:
-      "Public code search pattern. Literal by default; set regex: true to treat it as a regular expression.",
+      "Public code search pattern. Use literal code patterns as they appear in files, not keywords: e.g. `useState(` rather than `react hook state`.",
   }),
   regex: Type.Optional(
     Type.Boolean({
-      description:
-        "Treat pattern as a regular expression. Grep supports multi-line regex with (?s).",
+      description: "Treat `pattern` as a regular expression instead of literals.",
     }),
   ),
   repo: Type.Optional(
@@ -62,6 +61,7 @@ export const searchCodeTool = defineTool<typeof SearchCodeParams, SearchCodeDeta
   description: "Cross-repo code search over public source code.",
   promptSnippet: "Search code across repos",
   promptGuidelines: [
+    "When `regex` is true, `pattern` can also use `(?s)` for multi-line patterns.",
     "Results are CANDIDATES to verify via checkout_repo/read_github_file — never cite them directly.",
     "This searches public GitHub code only. Use search_github_code when private repository access matters.",
   ],

package/extensions/librarian/view.ts

@@ -1,21 +1,18 @@
 import path from "node:path";
 import type { AgentToolResult } from "@earendil-works/pi-agent-core";
 import type { Theme } from "@earendil-works/pi-coding-agent";
-import { getMarkdownTheme } from "@earendil-works/pi-coding-agent";
-import { Container, Markdown, Spacer, Text } from "@earendil-works/pi-tui";
+import { getMarkdownTheme, keyHint } from "@earendil-works/pi-coding-agent";
+import { type Component, Container, Markdown, Spacer, Text } from "@earendil-works/pi-tui";
 import type { LibrarianRunDetails, TraceCall } from "./run.ts";
 import { LIBRARIAN_TOOL_NAMES } from "./tools/names.ts";
 
 const COLLAPSED_TRACE_CALLS = 3;
+// Same frames and cadence as pi's Working... loader (pi-tui loader.ts defaults).
 const SPINNER_FRAMES = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
-const SPINNER_INTERVAL_MS = 80;
-
-interface LiveRenderContext {
-  invalidate(): void;
-  state: {
-    librarianSpinnerFrame?: number;
-    librarianSpinnerInterval?: ReturnType<typeof setInterval>;
-  };
+export const SPINNER_INTERVAL_MS = 80;
+
+interface LibrarianRenderContext {
+  lastComponent: Component | undefined;
 }
 
 export function formatDuration(milliseconds: number): string {
@@ -153,18 +150,17 @@ export function formatTraceLine(call: TraceCall, cacheDir: string): TraceLine {
   }
 }
 
-function renderTraceCallText(
-  call: TraceCall,
-  cacheDir: string,
-  theme: Theme,
-  spinnerFrame: string,
-): string {
+function currentSpinnerFrame(): string {
+  return SPINNER_FRAMES[Math.floor(Date.now() / SPINNER_INTERVAL_MS) % SPINNER_FRAMES.length] ?? "";
+}
+
+function renderTraceCallText(call: TraceCall, cacheDir: string, theme: Theme): string {
   const { verb, subject } = formatTraceLine(call, cacheDir);
   const running = call.endedAt === undefined;
   const icon = call.isError
     ? theme.fg("error", "✗")
     : running
-      ? theme.fg("warning", spinnerFrame)
+      ? theme.fg("accent", currentSpinnerFrame())
       : theme.fg("success", "✓");
   const duration = formatDuration((call.endedAt ?? Date.now()) - call.startedAt);
   const summary = call.resultSummary
@@ -189,6 +185,25 @@ function renderFooter(details: LibrarianRunDetails, theme: Theme): string {
   );
 }
 
+function collapsedFindingsHiddenText(details: LibrarianRunDetails): string | undefined {
+  const findings = details.findings;
+  if (!findings) {
+    return undefined;
+  }
+
+  const hiddenParts: string[] = [];
+  if (findings.locations.length > 0) {
+    hiddenParts.push(
+      `${findings.locations.length} location${findings.locations.length === 1 ? "" : "s"}`,
+    );
+  }
+  if (findings.description?.trim()) {
+    hiddenParts.push("details");
+  }
+
+  return hiddenParts.length > 0 ? `${hiddenParts.join(" and ")} hidden` : undefined;
+}
+
 function isLibrarianRunDetails(value: unknown): value is LibrarianRunDetails {
   return (
     value !== null &&
@@ -205,55 +220,33 @@ function renderTrace(
   expanded: boolean,
   cacheDir: string,
   theme: Theme,
-  spinnerFrame: string,
 ): string[] {
   const lines: string[] = [];
   const calls = expanded ? details.trace : details.trace.slice(-COLLAPSED_TRACE_CALLS);
   const hiddenCount = details.trace.length - calls.length;
   if (hiddenCount > 0) {
     lines.push(
-      theme.fg(
-        "muted",
-        `  … ${hiddenCount} earlier call${hiddenCount === 1 ? "" : "s"} (Ctrl+O to expand)`,
-      ),
+      theme.fg("muted", `  … ${hiddenCount} earlier call${hiddenCount === 1 ? "" : "s"} (`) +
+        keyHint("app.tools.expand", "to expand") +
+        theme.fg("muted", ")"),
     );
   }
   for (const call of calls) {
-    lines.push(renderTraceCallText(call, cacheDir, theme, spinnerFrame));
+    lines.push(renderTraceCallText(call, cacheDir, theme));
   }
   return lines;
 }
 
-function updateLiveRender(context: LiveRenderContext | undefined, running: boolean): string {
-  if (!context) {
-    return (
-      SPINNER_FRAMES[Math.floor(Date.now() / SPINNER_INTERVAL_MS) % SPINNER_FRAMES.length] ?? ""
-    );
-  }
-
-  context.state.librarianSpinnerFrame ??= 0;
-  if (running && !context.state.librarianSpinnerInterval) {
-    context.state.librarianSpinnerInterval = setInterval(() => {
-      context.state.librarianSpinnerFrame =
-        ((context.state.librarianSpinnerFrame ?? 0) + 1) % SPINNER_FRAMES.length;
-      context.invalidate();
-    }, SPINNER_INTERVAL_MS);
-  } else if (!running && context.state.librarianSpinnerInterval) {
-    clearInterval(context.state.librarianSpinnerInterval);
-    delete context.state.librarianSpinnerInterval;
-  }
-
-  return SPINNER_FRAMES[context.state.librarianSpinnerFrame] ?? "";
-}
-
 export function renderLibrarianResult(
   result: AgentToolResult<LibrarianRunDetails>,
   options: { expanded: boolean; isPartial: boolean },
   theme: Theme,
   cacheDir: string,
-  context?: LiveRenderContext,
+  context?: LibrarianRenderContext,
 ): Container {
-  const container = new Container();
+  const container =
+    context?.lastComponent instanceof Container ? context.lastComponent : new Container();
+  container.clear();
   const details = result.details;
 
   if (!isLibrarianRunDetails(details)) {
@@ -261,22 +254,34 @@ export function renderLibrarianResult(
     container.addChild(
       new Text(firstText && "text" in firstText ? firstText.text : "(no output)", 0, 0),
     );
+    container.invalidate();
     return container;
   }
 
   const running = options.isPartial || details.status === "running";
-  const spinnerFrame = updateLiveRender(context, running);
   container.addChild(new Text(renderQuestion(details, options.expanded, theme), 0, 0));
   container.addChild(new Spacer(1));
 
   if (running || !details.findings) {
-    for (const line of renderTrace(details, options.expanded, cacheDir, theme, spinnerFrame)) {
+    for (const line of renderTrace(details, options.expanded, cacheDir, theme)) {
       container.addChild(new Text(line, 0, 0));
     }
   }
 
   if (!running) {
     if (details.findings) {
+      const hiddenText = options.expanded ? undefined : collapsedFindingsHiddenText(details);
+      if (hiddenText) {
+        container.addChild(
+          new Text(
+            theme.fg("muted", `(${hiddenText}, `) +
+              keyHint("app.tools.expand", "to expand") +
+              theme.fg("muted", ")"),
+            0,
+            0,
+          ),
+        );
+      }
       const markdown = buildFindingsMarkdown(details, options.expanded);
       container.addChild(new Markdown(markdown, 0, 0, getMarkdownTheme()));
     } else if (details.error) {
@@ -290,6 +295,7 @@ export function renderLibrarianResult(
     container.addChild(new Text(theme.fg("muted", `run ${details.runId}`), 0, 0));
   }
   container.addChild(new Text(renderFooter(details, theme), 0, 0));
+  container.invalidate();
   return container;
 }
 

package/extensions/librarian.ts

@@ -17,6 +17,7 @@ import {
   formatTraceLine,
   renderLibrarianCall,
   renderLibrarianResult,
+  SPINNER_INTERVAL_MS,
   shorten,
 } from "./librarian/view.ts";
 
@@ -130,6 +131,16 @@ export default function librarianExtension(pi: ExtensionAPI): void {
     },
 
     renderResult(result, options, theme, context) {
+      // Live spinner/elapsed ticking, same pattern and cadence as pi's bash
+      // tool and Working... loader: re-render on an interval while streaming.
+      const state = context.state as { interval: NodeJS.Timeout | undefined };
+      if (options.isPartial && !state.interval) {
+        state.interval = setInterval(() => context.invalidate(), SPINNER_INTERVAL_MS);
+      }
+      if ((!options.isPartial || context.isError) && state.interval) {
+        clearInterval(state.interval);
+        state.interval = undefined;
+      }
       return renderLibrarianResult(result, options, theme, settings.cacheDir, context);
     },
   });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@thurstonsand/pi-librarian",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "type": "module",
   "description": "GitHub research subagent for pi: deep-dive specific repos, discover across the ecosystem",
   "license": "MIT",
@@ -25,7 +25,8 @@
     "CONTEXT.md",
     "DEV.md",
     "AGENTS.md",
-    "docs"
+    "docs",
+    "images"
   ],
   "pi": {
     "extensions": [