pi-context-engine 0.1.3 → 0.1.4

package/README.md

@@ -130,6 +130,16 @@ Environment overrides:
 - **`PCE_EMBED_BACKEND`**: force `local` or `ollama` instead of auto-detect.
 - **`PCE_RECALL_DISPLAY_CAP`**: how many recall hits to show (default 7).
 
+**Embedding backends**: two options, picked automatically per project.
+
+- **Ollama** (default when reachable): no extra model files on disk beyond the
+  Ollama model itself.
+- **Local ONNX** (transformers.js): works with no server. Needs the
+  `@huggingface/transformers` optional dependency, which plain `npm` and `pi`
+  installs include. If it was stripped (e.g. `--omit=optional`), indexing fails
+  with a clear message telling you to reinstall or switch to Ollama, instead of
+  a bare module error.
+
 ## Security
 
 Indexing is local, but the index is still a copy of your code. Two guards ship on

package/docs/IMPLEMENTATION-PLAN.md

@@ -0,0 +1,483 @@
+# Implementation plan: retrieval improvements
+
+Full build plan for the recommendations in
+[RESEARCH-RETRIEVAL.md](./RESEARCH-RETRIEVAL.md). Nothing here is started yet:
+implementation is deliberately delayed, this document defines exactly what will
+be built, in what order, and how each item is proven before its status changes.
+
+**Why a plan first:** every item changes retrieval behavior, and our only
+current metrics are token savings and latency. Building the measurement harness
+before the changes lets us prove each improvement instead of guessing.
+
+## How to use this document
+
+- **Status legend**
+  - **Not started**: planned, nothing built yet. All items are here today.
+  - **Ready**: implementation can begin (dependencies met, baseline recorded).
+  - **In progress**: work branch open.
+  - **Review**: code done, eval deltas attached to the PR.
+  - **Done**: merged, released, status table updated.
+  - **Dropped**: decided against, with the reason recorded in the item.
+- **Rules for flipping a status**
+  1. Typecheck and unit tests pass (`npm run check`).
+  2. Eval deltas (before/after) are recorded in the PR description.
+  3. README and `src/config.ts` cover any new config keys.
+  4. This table is updated in the same PR.
+- **Every item lists**: objective, evidence, files touched, steps, tests,
+  acceptance criteria, risks, dependencies, effort (S < half day, M 1-2 days,
+  L 3-5 days).
+
+---
+
+## Status summary
+
+| ID | Title | Phase | Priority | Effort | Status |
+|----|-------|-------|----------|--------|--------|
+| R4 | Labeled retrieval eval harness | 0 | P0 | M | Not started |
+| R1 | Contextual chunk headers | 1 | P0 | M | Not started |
+| R2 | BM25 phrases, column weights, symbol matches | 1 | P0 | S | Not started |
+| R3 | Query-aware compression | 1 | P0 | M | Not started |
+| R5 | Multi-path retrieval with symbol path | 2 | P1 | M | Not started |
+| R6 | Small-to-big context expansion | 2 | P1 | L | Not started |
+| R7 | Graph context packs + centrality prior | 2 | P1 | M | Not started |
+| R8 | Query-confidence signal + optional elaboration | 2 | P1 | M | Not started |
+| R9 | Refine suggestions in search results | 2 | P1 | S | Not started |
+| R10 | Opt-in local reranker | 3 | P2 | M | Not started |
+| R11 | Code-specialized embedder option | 3 | P2 | S | Not started |
+| R12 | AST density chunking (split-merge) | 3 | P2 | L | Not started |
+| R13 | Linked memory (A-MEM style) | 3 | P2 | M | Not started |
+
+**Blocked by nothing right now:** only R4 (phase 0). Everything else waits on
+R4's baseline so its impact can be measured.
+
+**Status history**
+- 2026-10-17: plan created from the research note. Implementation delayed by
+  owner decision. All items set to **Not started**.
+
+---
+
+## Phase 0: measurement foundation
+
+### R4. Labeled retrieval eval harness
+
+- **Objective:** measure retrieval quality (not just tokens) so every later
+  item has a before/after number.
+- **Evidence:** CORE-Bench (arXiv:2606.11864) shows code-search scores overstate
+  agentic usefulness; UDCG (arXiv:2510.21440) shows distracting chunks must be
+  counted; Voyage reports CoSQA has ~51% bad labels, so we label our own.
+- **Files:**
+  - new `tests/eval/queries.jsonl` (labels)
+  - new `scripts/eval-retrieval.mjs` (runner, tsx)
+  - `scripts/smoke-real-world.mjs` (add a `quality` block to its report)
+  - new `tests/eval/baseline.json` (recorded thresholds)
+  - `package.json` (`eval` script, wire into `check`)
+- **Steps:**
+  1. Write 60-100 labeled queries over this repo, four kinds: exact identifier
+     (`HybridRetriever`), natural language (`where is the retry logic`),
+     bug-report style (paste an error string), file-name style
+     (`src/retrieval/retriever.ts`). Each label lists expected file paths and,
+     where easy, expected chunk line ranges.
+  2. Runner indexes a fixture copy of the repo, runs each query through
+     `HybridRetriever`, and computes: Recall@5, Recall@10, nDCG@10, distraction
+     ratio (share of served chunks whose file is not in the label set), served
+     tokens, p50/p95 latency.
+  3. Implement nDCG in the script (no dependency); binary relevance is enough.
+  4. Record current numbers into `baseline.json`; the runner exits non-zero on
+     regression beyond a small tolerance (start at 2 points on Recall@10).
+  5. Add an optional external slice later (CORE-Bench level-2 sample) as a
+     separate non-gating script.
+- **Tests:** runner produces stable output twice in a row; a deliberately
+  broken query (random words) scores below baseline.
+- **Acceptance:** `npm run eval` reports all metrics, fails on regression, and
+  is documented in README's Development section.
+- **Risks:** labels drift when the repo changes. Mitigation: labels point at
+  files/paths, not exact line numbers, except where a test pins them.
+- **Dependencies:** none. **Effort:** M.
+
+---
+
+## Phase 1: quick wins (P0)
+
+### R1. Contextual chunk headers
+
+- **Objective:** give every chunk its surrounding context before embedding and
+  indexing, so "the retrieval module" style queries find `src/retrieval/*`
+  code.
+- **Evidence:** Anthropic Contextual Retrieval: failed retrievals −49% with
+  contextual embeddings + contextual BM25; late chunking (arXiv:2409.04701)
+  +2.7-3.6% nDCG from the same missing-context problem.
+- **Files:**
+  - `src/indexer/chunker.ts` (`nodeToChunk`, `walk`: collect enclosing class,
+    imports, exported symbol names)
+  - `src/types.ts` (optional `metadata.contextHeader`)
+  - `src/indexer/pipeline.ts` (`embedChunks` currently embeds `c.content`)
+  - `src/indexer/manifest.ts` (index format version bump)
+  - `src/indexer/embedding-cache.ts` (expect full cache invalidation)
+- **Steps:**
+  1. Define header format, deterministic and LLM-free:
+     `// file: src/retrieval/retriever.ts | lang: typescript | unit: class HybridRetriever | imports: ./confidence, ./query-parser`
+     Header must respect `indexer.redact_secrets` (path only, no file body).
+  2. Prepend the header in one place at chunk build time so embed, FTS and
+     `expand_chunk` all see the same text. Decision: header goes into
+     `content`, single source of truth, no second code path.
+  3. Bump `manifest` format version so existing installs reindex automatically
+     (content hash change already invalidates the embedding cache).
+  4. Keep header length capped (~120 tokens) so small chunks do not become
+     header-dominated.
+- **Tests:** header contains file path and unit name; cap respected; redacted
+  paths behave; ids stay unique (id hashes `content.slice(0,100)`, so the
+  header prefix changes ids: confirm full reindex, no stale rows).
+- **Acceptance:** eval shows Recall@k up or flat, no regression on identifier
+  queries, latency within 10% of baseline.
+- **Risks:** header noise on very small chunks; path-only leakage of private
+  directory names (accepted, index is local and paths already go to FTS).
+- **Dependencies:** R4 baseline. **Effort:** M.
+
+### R2. Strengthen BM25: phrases, column weights, symbol matches
+
+- **Objective:** make the strongest code retriever work properly for quoted
+  phrases and path queries.
+- **Evidence:** Finding 6 (BM25 strongest for code, arXiv:2605.14503);
+  Anthropic reports contextual BM25 as the larger single jump (35% → 49%).
+- **Files:** `src/storage/fts-store.ts` (`escapeFts5`, `search`),
+  `src/retrieval/retriever.ts` (existing `FTS_BOOST_CODE_LOOKUP`),
+  tests under `tests/`.
+- **Steps:**
+  1. Parse quoted spans before tokenizing; emit FTS5 phrase strings
+     (`"foo bar"`) for them and keep OR-joining unquoted tokens.
+  2. Add explicit `bm25()` column weights in `ORDER BY`: weight `file_path`
+     highest, then `content`, then `chunk_type`; leave `language` neutral.
+  3. Exact symbol shortcut: if a query token matches a chunk's symbol name
+     (metadata from R1/R12, fallback regex on content), give it a deterministic
+     score bonus in `retriever.ts`.
+  4. Keep stopword fallback and the `code_lookup` FTS weight as they are.
+- **Tests:** phrase query matches only adjacent terms; path query ranks its
+  file top-3; OR behavior for unquoted terms unchanged; empty/stopword-only
+  queries still return the existing fallback.
+- **Acceptance:** identifier and file-name slices of the eval improve; no
+  slice regresses by more than 1 point.
+- **Risks:** FTS5 phrase syntax with punctuation-heavy tokens (code symbols
+  contain `_`, `.`); mitigate by tokenizing phrase content the same way FTS5
+  tokenizer does for indexed content.
+- **Dependencies:** R4. **Effort:** S.
+
+### R3. Query-aware compression
+
+- **Objective:** stop cutting the lines that answer the question.
+- **Evidence:** Finding 12 (aggressive compression consistently harmful,
+  budget ≥ 8000, arXiv:2605.14503); SWE-Pruner (arXiv:2601.16746) 23-54% token
+  cut with task-conditioned selection; LLMLingua budget controller.
+- **Files:** `src/compression/compressor.ts` (`TRUNCATION_LIMITS`, prompts),
+  `src/retrieval/retriever.ts` (packing uses
+  `chunk.compressedContent` length), `src/config.ts`, README config block.
+- **Steps:**
+  1. Add `compress(content, level, query?)`. With a query: keep signature
+     lines, import block, any line containing a query token or identifier, then
+     apply the cap to the remainder.
+  2. Pass query terms into the Ollama summary prompt (`{query}` slot) so
+     summaries mention the asked-for symbols; keep the existing
+     `QualityChecker` identifier-ratio gate as the safety net.
+  3. `full` becomes budget-driven: no fixed cap, only the retrieval token
+     budget trims it (aligns with the ≥ 8000 guidance).
+  4. New config: `compression.query_aware: true` (default on), so users can
+     revert to the old fixed behavior.
+- **Tests:** query tokens always survive truncation; identifier ratio stays
+  ≥ 0.4; without query, output is byte-identical to today (regression guard).
+- **Acceptance:** at equal or lower served tokens, Recall@k does not drop; on
+  the bug-report slice it rises.
+- **Risks:** keeping many matched lines can exceed caps; cap logic must prefer
+  matched lines over head lines, with a hard ceiling.
+- **Dependencies:** R4. **Effort:** M.
+
+---
+
+## Phase 2: retrieval depth (P1)
+
+### R5. Multi-path retrieval with a structured symbol path
+
+- **Objective:** stop relying on similarity alone for "where is X defined".
+- **Evidence:** CodeRAG (arXiv:2509.16112) multi-path retrieval is a core gain;
+  the coding-agent preprint found production agents mix lexical, semantic and
+  structural signals.
+- **Files:** `src/storage/graph-store.ts` (symbol lookup), `src/retrieval/
+  retriever.ts` (third RRF list), `src/retrieval/query-parser.ts` (extract
+  symbol candidates), `src/storage/local-backend.ts`.
+- **Steps:**
+  1. Add `findSymbol(name)`: exact then case-insensitive then prefix match
+     over graph node names (function/class/file nodes).
+  2. Add file-path exact match as its own path using existing `fileHints`.
+  3. Fuse vector, FTS and symbol results with the existing RRF (k=60); symbol
+     and path lists get the same weight class as `code_lookup`'s FTS boost.
+  4. Keep `MAX_CHUNKS_PER_FILE` and marginal-ratio logic after fusion.
+- **Tests:** exact symbol query returns its chunk in top-3; unknown symbol
+  falls back to current behavior; fusion does not double-count a chunk.
+- **Acceptance:** identifier slice Recall@5 improves; general slice flat.
+- **Risks:** symbol collisions across files (dedupe by file, prefer file hint).
+  **Dependencies:** R4, benefits from R1 headers. **Effort:** M.
+
+### R6. Small-to-big context expansion
+
+- **Objective:** serve enough surrounding context per hit to answer without a
+  second tool call.
+- **Evidence:** cross-file context length was the dominant parameter
+  (2048 → 8192 tokens = up to +4.2 points, arXiv:2605.04763); small-to-big
+  retrieval is the standard fix; cAST metadata retention added +2.7 on SWE-bench.
+- **Files:** `src/indexer/chunker.ts` (emit module/top-level chunks),
+  `src/retrieval/retriever.ts` (expansion stage, packing),
+  `src/config.ts`.
+- **Steps:**
+  1. Emit, for every AST file, a `module` chunk covering top-level statements,
+     imports and exported names, so code outside functions is retrievable.
+  2. After ranking, expand each top hit: attach the file's module chunk plus
+     one neighboring chunk above/below when budget allows. Mark provenance in
+     metadata (`served_via: self|module|neighbor`) for eval analysis.
+  3. Expansion must run inside the `maxTokens` budget and must not break the
+     marginal-ratio stop (expanded context counts as part of the parent hit).
+  4. Fix the interaction with `dedupeOverlaps`: a module chunk overlaps every
+     function chunk in the file and would be dropped. Exempt parent-tier chunks
+     from overlap dedupe, dedupe only within a tier.
+- **Tests:** module chunk survives dedupe; expansion respects budget;
+  provenance recorded; concat property of module + function chunks still
+  reconstructs the file.
+- **Acceptance:** cross-file and NL slices improve Recall@10 and distraction
+  ratio, with served tokens up by no more than the configured budget share.
+- **Risks:** token creep; duplicate content between module and function chunks
+  wastes budget (mitigate: serve module chunk in compressed form by default).
+  **Dependencies:** R4, R3 (compression runs on expanded content). **Effort:** L.
+
+### R7. Graph context packs with a centrality prior
+
+- **Objective:** make the repository graph a real retrieval path, not a
+  2-chunk bonus.
+- **Evidence:** RepoGraph (ICLR 2025) gained +2.66/+2.34 absolute on
+  SWE-bench-Lite across four systems at roughly constant cost; Aider's PageRank
+  file ranking was observed working in practice.
+- **Files:** `src/storage/graph-store.ts`, `src/storage/local-backend.ts`,
+  `src/indexer/pipeline.ts` (compute prior at index time),
+  `src/retrieval/retriever.ts`, `src/extension.ts` (`related_context` output).
+- **Steps:**
+  1. At index time, run a small power-iteration PageRank over the file-level
+     import graph and store it as node `properties.pagerank`.
+  2. In retrieval, build a context pack per top hit: the node's callers,
+     callees, importers and importees with edge types and line refs, capped at
+     N lines and included in the token budget.
+  3. Apply centrality as a bounded prior: score multiplier in
+     `[1 - p, 1 + p]` with `retrieval.graph_prior` default `0.05`. Never let it
+     outrank a relevance gap larger than the prior.
+  4. Extend `related_context` to return the same packs for manual drilling.
+- **Tests:** pack contains at least one caller/importer on a fixture repo;
+  prior is clamped to its range; graph failures stay best-effort (existing
+  try/catch behavior).
+- **Acceptance:** cross-file slice improves; cost per query stays within
+  +20 ms of baseline warm latency.
+- **Risks:** dense graphs flood the pack (cap by degree, prefer higher-ranked
+  neighbors); PageRank staleness on incremental reindex (recompute per
+  full index, reuse on partial).
+- **Dependencies:** R4, R6 (packs ride the same budget path). **Effort:** M.
+
+### R8. Query-confidence signal and optional elaboration
+
+- **Objective:** tell the agent when a search was weak, and let it recover,
+  without silent query rewriting.
+- **Evidence:** predicted query performance correlates with answer quality
+  (arXiv:2507.10411); Findings 1/2/4 (raw query wins, elaborate only when
+  ambiguous, LLM rewrites drift); Search-R1 for the loop framing.
+- **Files:** `src/retrieval/confidence.ts`, `src/retrieval/query-parser.ts`,
+  `src/guidance.ts`, `src/extension.ts` (tool result shape), `src/config.ts`.
+- **Steps:**
+  1. Compute query-level signals: top final score, margin to the next distinct
+     file, vector/FTS agreement (Jaccard of their top-10), candidate count.
+  2. Map to `query_confidence: high|medium|low` with thresholds calibrated
+     against the eval set, stored in config so they are tunable.
+  3. Return the signal plus 1-3 deterministic recovery hints (add a file path,
+     use an exact symbol, drop vague terms) when confidence is low.
+  4. Optional elaboration behind `retrieval.auto_elaborate` (default `false`):
+     generate a deterministic variant (stopword-stripped, file hints injected,
+     terms from near-miss top hits), run both, fuse with RRF, always keep the
+     raw query's results as primary.
+  5. Document the signal and the loop in the guidance section so the agent
+     knows to refine instead of re-reading files.
+- **Tests:** weak synthetic query gets `low` and hints; strong identifier query
+  gets `high`; elaboration never removes raw-query results; no network calls.
+- **Acceptance:** guidance renders, config off switches everything off, eval
+  unchanged when elaboration is disabled.
+- **Risks:** noisy confidence on tiny indexes (calibrate on the fixture, floor
+  the candidate count). **Dependencies:** R4. **Effort:** M.
+
+### R9. Refine suggestions in search results
+
+- **Objective:** turn a one-shot tool into a guided loop at near-zero token cost.
+- **Evidence:** every production coding agent in the exploratory preprint
+  narrows iteratively; RepoCoder's loop drives its reported gain.
+- **Files:** `src/extension.ts` (result shaping), optionally
+  `src/engine.ts`; guidance text in `src/guidance.ts`.
+- **Steps:**
+  1. From the top hits, derive up to 3 next queries: a sibling symbol from the
+     same file (graph `defines` edges), a near-miss file ranked 11-15 that was
+     not served, one alternate term that appears often in top hits.
+  2. Emit as a single `refine:` line in the tool result; cap length.
+  3. Config `retrieval.refine_hints` (default `true`), documented in README.
+- **Tests:** cap of 3; never suggests a file already served; disabled by config.
+- **Acceptance:** token overhead ≤ 40 tokens per call; no eval regression.
+- **Risks:** hints duplicating what the agent would do anyway (measure usage in
+  session telemetry later). **Dependencies:** R5, R7 for better symbol/family
+  hints (can ship with weaker heuristics first). **Effort:** S.
+
+---
+
+## Phase 3: gated experiments (P2)
+
+Each phase-3 item ships **behind config, default off**, and only graduates to
+default-on if the eval shows a win and latency budget holds.
+
+### R10. Opt-in local reranker
+
+- **Objective:** test whether cross-encoder reranking helps *our* workloads.
+- **Evidence is split, on purpose:** harmful for SE tasks with 815 ms GPU cost
+  (Findings 10/11), a core gain in CodeRAG, −67% failures with Anthropic's
+  stack. Only our eval can decide.
+- **Files:** new `src/retrieval/reranker.ts`, `src/config.ts`, README.
+- **Steps:**
+  1. Use the already-shipped `@huggingface/transformers` for
+     `text-classification` with an ONNX reranker (candidate:
+     `Xenova/ms-marco-MiniLM-L-6-v2`), cached in the existing models dir.
+  2. Rerank only the fused candidate list (cap 30), then re-apply path penalty,
+     file diversity and marginal stop.
+  3. Config: `retrieval.rerank: off | local` (default `off`),
+     `retrieval.rerank_model`, `retrieval.rerank_topn`.
+  4. One-time model download must be explicit (never on by default) because it
+     adds ~80 MB to a package that markets "one npm command".
+- **Tests:** mock pipeline ordering (no model in unit tests); opt-in path
+  exercised in a smoke test with a stubbed model.
+- **Acceptance:** measured eval delta (up or down) and p95 latency recorded in
+  the PR; default stays `off` unless quality wins justify the download.
+- **Risks:** CPU latency; ONNX export availability for the chosen model.
+  **Dependencies:** R4 mandatory. **Effort:** M.
+
+### R11. Code-specialized embedder option
+
+- **Objective:** offer, and only recommend, a code-trained local embedder.
+- **Evidence:** Findings 7/8 say scale and code-specialization did not decide
+  outcomes, but CORE-Bench shows agentic retrieval is harder and in-domain
+  training helped a lot.
+- **Files:** `src/indexer/embedder.ts` (model list), README embedding section.
+- **Steps:**
+  1. Document `embedding.model: <hf-id>` with one tested code-oriented model
+     and the existing general defaults.
+  2. Run the eval for each backend; publish the table in README.
+  3. Dimension change already forces full reindex; verify the migration path.
+- **Tests:** backend selection unit test; reindex-on-dimension-change test
+  (logic exists, add a regression test if missing).
+- **Acceptance:** README shows measured numbers per model, not claims.
+- **Dependencies:** R4. **Effort:** S.
+
+### R12. AST density chunking (cAST-style split-merge)
+
+- **Objective:** replace worst-measured strategy (whole function nodes) with
+  budgeted, self-contained chunks.
+- **Evidence:** function chunking −3.57 to −5.64 points and never
+  Pareto-optimal (arXiv:2605.04763); cAST +4.3 Recall@5, +2.67 Pass@1
+  (arXiv:2506.15655).
+- **Files:** `src/indexer/chunker.ts` (core rework), `src/types.ts`
+  (metadata: `symbol`, `parent`, `segment`), `src/config.ts`, chunk-id scheme
+  in `makeChunk`.
+- **Steps:**
+  1. Top-down fit: a node fits if its non-whitespace chars ≤
+     `indexer.chunk_budget_chars` (default ~1200, configurable).
+  2. Oversized nodes split at syntactic children; only leaf-level overflow uses
+     line windows. Greedy-merge small adjacent siblings up to budget.
+  3. Emit module/top-level chunks (shared with R6).
+  4. Preserve chunk-id uniqueness for split segments (add segment index; keep
+     the upstream column-offset fix that the README documents).
+  5. Keep the legacy chunker behind `indexer.chunking: legacy | ast-density`
+     (default `legacy`) for A/B on the same index format.
+  6. Verify the concat property: chunks concatenated reproduce the file.
+- **Tests:** concat round-trip; id uniqueness on duplicate spans; budget
+  respected; legacy mode byte-identical to today.
+- **Acceptance:** `ast-density` wins the eval on at least Recall@10 without
+  inflating chunk count beyond +30%; then flip the default in its own release.
+- **Risks:** more chunks = more embeddings = slower first index (content-hash
+  cache limits repeat cost); id scheme changes force a full reindex.
+  **Dependencies:** R4, R6 (module chunks shared), ideally after R1 so headers
+  apply to the new chunk shapes. **Effort:** L.
+
+### R13. Linked memory (A-MEM style)
+
+- **Objective:** make cross-session recall improve with age instead of
+  degrading.
+- **Evidence:** A-MEM (NeurIPS 2025): dynamic indexing, linking between
+  memories and memory evolution beat fixed structures on six models.
+- **Files:** `src/memory/db.ts`, `decision-extractor.ts`,
+  `session-capture.ts`, `record_decision` tool surface in `extension.ts`.
+- **Steps:**
+  1. On `record_decision`: embed it, find top-3 past decisions above a
+     similarity threshold, store `related_to` edges (new edge type alongside
+     `decided`/`discussed_in`), cap links per memory.
+  2. Store structured attributes on write: keywords, tags, affected files
+     (files already feed `record_code_area`).
+  3. Memory evolution: allow a later decision to annotate an earlier one
+     (`relatedTo` tool parameter) instead of only appending rows; keep the
+     original text immutable, append annotations with timestamps.
+  4. Surface links in `session_recall` output so the agent can walk them.
+- **Tests:** threshold and cap behavior; no self-links; redaction still applies
+  before any write (PII guard must run before link text is stored).
+- **Acceptance:** a seeded recall test shows linked neighbours surfacing;
+  latency of `record_decision` stays under +100 ms.
+- **Risks:** link noise from loose thresholds (calibrate on a fixture set).
+  **Dependencies:** R4-style eval extended to memory recall. **Effort:** M.
+
+---
+
+## Cross-cutting work
+
+- **Index format versioning:** R1, R6 and R12 each change on-disk content.
+  Add `manifest.formatVersion`; bump it per item and force a full reindex with
+  a logged migration line (the dimension-change path in `pipeline.ts` is the
+  template).
+- **Config surface:** new keys land in `src/config.ts`, the README
+  Configuration block, and the guidance text in one PR per item. Planned keys:
+  - `compression.query_aware`, `retrieval.auto_elaborate`,
+    `retrieval.graph_prior`, `retrieval.refine_hints`,
+    `retrieval.rerank`, `retrieval.rerank_model`, `retrieval.rerank_topn`,
+    `indexer.chunk_budget_chars`, `indexer.chunking`.
+- **Guidance updates:** R8 and R9 change how the agent should use the tool, so
+  `src/guidance.ts` must document the confidence signal and refine hints in the
+  same release, keeping the ~900-token budget.
+- **Docs:** README tables (What you get, How it works, Configuration,
+  Differences from upstream) updated per release; research evidence lives in
+  `RESEARCH-RETRIEVAL.md`, build detail here.
+- **Releases:** every item ships through a PR to `main` and a `v*.*.*` tag per
+  the packages AGENTS.md recipe. Suggested grouping, adjustable once baselines
+  exist:
+  - **0.2.0**: R4 + R1 + R2 + R3 (measurement + phase 1), format version bump.
+  - **0.3.0**: R5 + R6 + R7 (retrieval depth).
+  - **0.4.0**: R8 + R9 (agentic loop).
+  - **0.5.0+**: whichever of R10-R13 wins its experiment, one per release.
+- **Definition of done (per item):** `npm run check` green, eval deltas in the
+  PR, README/config/guidance updated, status table in this file flipped.
+
+---
+
+## Explicit non-goals
+
+Recorded so they are not re-litigated later:
+
+- **No always-on LLM query rewriting** (Finding 4, +987 tokens per rewrite).
+- **No default reranking** (Findings 10-11; R10 stays opt-in).
+- **No aggressive compression defaults** (Finding 12).
+- **No chunk overlap** by default (two evaluations found no clear benefit).
+- **No bigger-embedder reflex** (Findings 7-8); R11 must win on the eval.
+- **No full-file indexing** as a substitute for retrieval (Anthropic notes it
+  only makes sense under ~200k tokens).
+
+---
+
+## Open questions that gate work
+
+1. Does the function-chunking penalty transfer to our question workload?
+   R4 then R12 answer this with data.
+2. Which side of the reranking debate applies to pi? R10 answers it.
+3. Should confidence weights (0.5 vector / 0.4 keyword / 0.1 recency) be
+   fitted on the eval set instead of fixed? A cheap follow-up after R4.
+4. Do refine suggestions change agent behavior in real sessions? Needs opt-in
+   session telemetry after R9.

package/docs/RESEARCH-RETRIEVAL.md

@@ -0,0 +1,532 @@
+# Research: code retrieval literature → improvement backlog
+
+This note summarizes what peer-reviewed and preprint research says about code
+retrieval and code RAG, and maps each finding onto pi-context-engine.
+
+**Why it exists:** we currently optimize for token savings and latency, both of
+which we measure. Retrieval *quality* we do not measure. The literature shows
+that retriever-side choices dominate final answer quality, so this is the
+highest-leverage area to study before adding features.
+
+**How to read it:**
+- **Evidence A**: primary source read in full or abstract (arXiv/ACL/official blog).
+- **Evidence B**: found via search result snippet only, not yet verified in full.
+- **Recommendations** are tagged **P0** (quick win), **P1** (next), **P2**
+  (experiment, gate behind config). Every one lists the files to touch and how
+  to prove it worked.
+
+Research done 2026-10-17. Sources are listed at the end with links.
+
+**Implementation:** deferred by owner decision. The build plan, with per-item
+steps, tests, acceptance criteria and a status table, lives in
+[IMPLEMENTATION-PLAN.md](./IMPLEMENTATION-PLAN.md). Nothing is started yet.
+
+---
+
+## 1. What the evidence says
+
+### 1.1 The retriever matters more than the generator
+
+- **Finding**: in a component-wise study of 21+ models across code generation,
+  summarization and repair, retriever-side components (query processing,
+  retrieval algorithm, context refinement) influenced final performance more
+  than the choice of generator model.
+- **Why it matters for us:** pi-context-engine *is* the retriever side. Every
+  improvement here lands on every model the user runs, cheap and model-agnostic.
+- **Evidence A**: Not All RAGs Are Created Equal (arXiv:2605.14503).
+
+### 1.2 BM25 is remarkably strong on code; hybrid is safe but conservative
+
+- **Finding 6**: BM25 was "an exceptionally powerful and versatile retriever"
+  across text-heavy, clean and noisy code queries, often beating larger dense
+  models. Exact matches on algorithm names and error strings drive this.
+- **Finding 9**: hybrid retrieval was "safe but conservative": in their
+  experiments it did **not** beat the single best retriever for a given task.
+- **Finding 7/8**: bigger embedding models did not win, and code-specific
+  training did not reliably beat strong general-purpose models.
+- **Why it matters for us:** our RRF hybrid and small general embedders
+  (all-MiniLM-L6-v2, nomic-embed-text) are consistent with the evidence. The
+  win is not a bigger model, it is better lexical handling and better query
+  understanding.
+- **Evidence A**: arXiv:2605.14503, Findings 5-9.
+
+### 1.3 Query rewriting: keep the raw query by default
+
+- **Finding 1**: the original, unaltered query is a powerful baseline and often
+  beats automated transformations when the task input is clear.
+- **Finding 2**: elaboration helps only for ambiguous or noisy inputs.
+- **Finding 4**: every LLM-based rewrite risks changing intent or dropping
+  critical details ("semantic drift").
+- **Cost note**: an elaborated query cost +987 input tokens, roughly the price
+  of 2-3 extra retrieved documents.
+- **Why it matters for us:** our `QueryParser` does regex intent
+  classification and keyword extraction, which is free and drift-free. The
+  evidence says do not add LLM query rewriting as a default.
+- **Evidence A**: arXiv:2605.14503, Findings 1-4 and the cost table.
+
+### 1.4 Reranking: contested evidence, do not default to it
+
+- **Negative (SE tasks)**: Findings 10/11 report reranking as "unreliable" and
+  "often detrimental": it disrupted an already good BM25 ordering and failed to
+  rescue weak retrievals. Their cost table shows a BGE reranker at 815 ms on a
+  12.8 GB GPU, versus <10 ms for BM25.
+- **Positive (repo-level completion)**: CodeRAG's preference-aligned "BestFit"
+  reranking (LLM reranker, then distilled to a small model) is a core reason it
+  beats SOTA on ReccEval and CCEval.
+- **Positive (general RAG)**: Anthropic reports failed retrievals drop by 67%
+  when reranking is added on top of contextual retrieval.
+- **Positive (code search)**: two-stage retrieve-then-rerank work reports
+  consistent gains for text-to-code search (arXiv:2208.11274, Evidence B snippet).
+- **Why it matters for us:** the sign of the effect depends on the task. A
+  reranker is a reasonable *opt-in experiment*, never a default, until our own
+  eval says otherwise.
+- **Evidence A**: arXiv:2605.14503 (Findings 10-11), arXiv:2509.16112,
+  Anthropic contextual retrieval blog.
+
+### 1.5 Compression: gentle and task-aware helps, aggressive hurts
+
+- **Finding 12**: compression utility depends on the token budget and the
+  compressor. Code-specialized compressors with a generous budget were robust;
+  aggressive compression was consistently harmful. Their advice: if you compress,
+  keep a budget of at least 8000 tokens.
+- **SWE-Pruner** (2026): a goal-conditioned 0.6B "skimmer" that keeps task-relevant
+  lines achieved 23-54% token reduction on SWE-bench Verified *while improving*
+  success rates, and up to 14.84x on single-turn QA. The key idea: pruning is
+  conditioned on the current task goal, not a fixed rule.
+- **LLMLingua** (EMNLP 2023): a budget controller plus perplexity-based token
+  dropping reaches up to 20x compression with little loss.
+- **Provence** (ICLR 2025): a lightweight trained pruner removes irrelevant
+  sentences from retrieved contexts before generation.
+- **Why it matters for us:** our compressor truncates at fixed character caps
+  (100/300/800 by level) with no knowledge of the query, and our LLM summarizer
+  prompt is query-agnostic. Fixed caps can cut exactly the lines the agent needs.
+- **Evidence A**: arXiv:2605.14503, arXiv:2601.16746, arXiv:2310.05736,
+  arXiv:2501.16214.
+
+### 1.6 Chunking: pure function-sized AST chunks measured worst
+
+- **Controlled study (864 configurations, RepoEval + CrossCodeEval)**:
+  function-based chunking underperformed every other strategy by 3.57-5.64
+  percentage points (Cliff's delta = -1.0). Sliding window and cAST sat on the
+  cost-quality Pareto front; function chunking was never Pareto-optimal.
+- **Dominant parameter**: cross-file context *length*. Doubling from 2048 to
+  8192 tokens gave up to +4.2 points, while chunk size had a weaker,
+  non-monotonic effect.
+- **cAST** (AST split-then-merge chunking): +4.3 Recall@5 on RepoEval retrieval,
+  +2.67 Pass@1 on SWE-bench. It packs AST nodes to a size budget measured in
+  non-whitespace characters, merges small siblings, and preserves file/class/
+  function metadata so hybrid matching improves.
+- **Overlap**: two sources agree overlap is not clearly helpful (late chunking
+  appendix; a 2601 chunking study says "avoid overlap").
+- **Why it matters for us:** `chunker.ts` emits whole functions and classes
+  exactly as parsed. There is no size budget, no sibling merge, no parent
+  context, and top-level code outside functions is not its own chunk. Our tasks
+  are questions rather than completion, so the transfer is partial, but the
+  direction is clear: density and surrounding context beat raw node boundaries.
+- **Evidence A**: arXiv:2605.04763, arXiv:2506.15655, arXiv:2409.04701.
+
+### 1.7 Contextualized chunks: the biggest cheap win in the literature
+
+- **Anthropic Contextual Retrieval**: prepend a short chunk-specific context
+  statement before embedding and before building the BM25 index.
+  - Contextual embeddings alone: top-20 failure rate 5.7% → 3.7% (−35%).
+  - Embeddings + contextual BM25: 5.7% → 2.9% (−49%).
+  - Plus reranking: −67%.
+- **Late chunking** (Jina): embed the whole document first, pool per chunk
+  afterwards. +2.7% to +3.63% relative nDCG across models and datasets, no extra
+  training, and it beats naive chunking in almost every configuration.
+- **Why it matters for us:** `pipeline.ts` embeds `c.content` verbatim. A chunk
+  from `src/retrieval/retriever.ts` carries no clue that it lives in the
+  retrieval module unless the words appear in the code itself. Deterministic
+  context headers (path, enclosing class, sibling names, imports) are free, need
+  no LLM, and match what both papers show helps most.
+- **Evidence A**: Anthropic engineering blog, arXiv:2409.04701.
+
+### 1.8 Repository structure and graphs pay off
+
+- **RepoGraph** (ICLR 2025): an AST-built graph (reference/definition nodes,
+  invoke/contain edges) plugged into four systems raised SWE-bench-Lite resolve
+  rate by +2.66 and +2.34 absolute, at roughly proportional or lower cost. Best
+  combined result was 29.67, state of the art among open-source methods at the time.
+- **Aider** ranks context files with PageRank over the repo graph (cited in the
+  RepoGraph related work), which our own preprint source also observed in practice.
+- **Why it matters for us:** we already build `calls`/`imports`/`defines` edges,
+  but graph expansion in `retriever.ts` only adds 2 bonus chunks from 2 related
+  files with a fixed 0.85 score factor. There is headroom to make the graph a
+  first-class retrieval path.
+- **Evidence A**: arXiv:2410.14684.
+
+### 1.9 Real coding agents retrieve iteratively and along several paths
+
+- **RepoCoder** (EMNLP 2023): iterative retrieval then generation, rerunning
+  retrieval with the model's own partial output, improved repository-level
+  completion by over 10% (Evidence B for the number, A for the mechanism).
+- **CodeRAG** (EMNLP 2025): attacks three failure modes directly: bad query
+  construction (uses log-probability guided probing instead of "last k lines"),
+  single-path retrieval (runs several retrieval paths and fuses), and
+  retriever/generator mismatch (reranks by generator preference).
+- **Exploratory study of code retrieval in coding agents** (2025 preprint, not
+  peer reviewed): across Claude Code, Codex CLI, Gemini CLI, Cursor, Amp, Cline
+  and Aider, every agent used progressive narrowing: broad first query, then
+  tighter follow-ups. It also found agents mixing lexical, semantic, LSP and
+  graph-based signals rather than relying on one.
+- **Query performance prediction in agentic RAG** (2025): predicted quality of
+  the generated query correlates positively with final answer quality, and
+  better retrievers produce higher quality answers in shorter reasoning chains.
+  This is the basis for adaptive retrieval: signal low confidence and retry.
+- **Why it matters for us:** our guidance tells the agent how to phrase one
+  query. The literature says the winning pattern is a *loop*: search, judge,
+  refine. We can support that with confidence signalling and refine hints.
+- **Evidence A**: arXiv:2303.12570 (mechanism), arXiv:2509.16112,
+  arXiv:2507.10411; Evidence B: preprint 10.20944/preprints202510.0924.
+
+### 1.10 Evaluation: public code-search benchmarks mislead; measure recall and distraction yourself
+
+- **CORE-Bench** (2026): agentic coding retrieval is "requirement-driven
+  repository search", not snippet matching. Representative embedding models
+  show a *sharp drop* from traditional code search to this setting, because of
+  intent-to-implementation gaps, scattered evidence, long chunks that dilute the
+  embedding, and dense in-repo distractors.
+- **Voyage AI**: CoSQA has an estimated 51% incorrect labels, CodeSearchNet
+  queries are copied from docstrings (too easy), and public sets suffer data
+  contamination. They recommend mining eval queries from real SWE-bench-style work.
+- **Metric mismatch**: UDCG (2025) shows classical nDCG/MAP/MRR mispredict RAG
+  performance because they ignore *distracting* (related but useless) documents;
+  a utility-and-distraction-aware metric improved correlation with end-to-end
+  accuracy by up to 36%.
+- **Why it matters for us:** `smoke:real-world` measures tokens and latency only.
+  We cannot tell whether a change helps or hurts retrieval quality today, and
+  our 97% savings claim has no paired quality number.
+- **Evidence A**: arXiv:2606.11864, arXiv:2510.21440, Voyage AI blog.
+
+### 1.11 Memory: link memories, do not just store them
+
+- **A-MEM** (NeurIPS 2025): Zettelkasten-style memory with dynamic indexing and
+  linking. New memories get structured attributes (context, keywords, tags),
+  get linked to related old memories, and can *update* older memories as
+  understanding improves. Beats SOTA memory baselines on six foundation models.
+- **MemGPT** (2023): memory as an OS-style paging system, the reference frame
+  for tiered agent memory.
+- **Why it matters for us:** `record_decision` stores items that are searchable
+  but not linked to each other or to the code areas they concern, and old
+  entries are never revised.
+- **Evidence A**: arXiv:2502.12110, arXiv:2310.08560.
+
+---
+
+## 2. Gap analysis: pi-context-engine today vs the evidence
+
+| Area | Today | Evidence says | Gap |
+|---|---|---|---|
+| Chunk embedding text | raw `chunk.content` only | contextual headers cut failed retrievals by 49% | no path/class/import context in vectors or FTS |
+| Chunking | whole functions/classes, no size budget | function chunking measured worst; split-merge + density wins | no split-merge, no sibling merge, no top-level code chunks |
+| Served context length | packed to token budget, one chunk at a time | cross-file context length is the dominant quality factor | no parent/neighbor expansion around a hit |
+| Compression | fixed 100/300/800 char caps, query-agnostic summaries | gentle, task-conditioned pruning helps; aggressive hurts | compression ignores the query |
+| FTS/BM25 | OR-joined tokens, quoted phrases flattened, default column weights | BM25 strongest single retriever for code; contextual BM25 gave the bigger jump | no phrase search, no path/file column weighting, no symbol path |
+| Query handling | regex intents + keywords, free and drift-free | raw query best; elaborate only when ambiguous; QPP signals help | no low-confidence feedback loop, no refine hints |
+| Graph | 1-hop file bonus: 2 files × 2 chunks, fixed 0.85 factor | RepoGraph: +2.66/+2.34 absolute SWE-bench resolve | graph is a footnote, not a retrieval path |
+| Reranking | none | contested: negative for SE tasks, positive for repo completion | fine as-is; only revisit as a gated experiment |
+| Evaluation | tokens + latency | recall + distraction metrics needed to trust any change | no labeled query set, no quality metric |
+| Memory | stored + searchable, no links | linking and memory evolution improve recall | no decision-to-decision or decision-to-code links |
+
+---
+
+## 3. Recommendations
+
+### P0: quick wins, no new dependencies
+
+**R1. Prepend deterministic context headers to chunks (indexing + FTS).**
+- **Do:** build a header per chunk: file path, language, enclosing class or
+  module, function signature, imports of the file, and a one-line "part of
+  file X" statement. Prepend it before embedding *and* before FTS ingest.
+- **Why:** Anthropic measured a 49% cut in failed retrievals from exactly this,
+  and late chunking shows the same "chunks lose surrounding context" effect
+  (+2.7-3.6% nDCG). Our headers need no LLM, so there is no per-chunk cost.
+- **Where:** `src/indexer/chunker.ts` (`nodeToChunk`), `src/indexer/pipeline.ts`
+  (`embedChunks` embeds `c.content`), `src/storage/fts-store.ts` (`ingest`).
+  The embedding cache is keyed by content hash, so the header change triggers a
+  clean full re-embed; bump an index-format version to make that explicit.
+- **Verify:** Recall@k on the labeled set (R4) before/after, plus a canary
+  query per file that mentions the module name but not the code words.
+
+**R2. Strengthen BM25: phrases, column weights, symbol matching.**
+- **Do:** detect quoted input and keep it as an FTS5 phrase; weight `file_path`
+  and `chunk_type` columns explicitly in the `bm25()` call; surface exact
+  symbol hits (a query token matching a function/class name) as a strong signal.
+- **Why:** BM25 is the most reliable retriever for code in the study, and exact
+  identifiers are where dense embeddings are weakest. We already OR-join tokens,
+  which is safe, but phrases like `"ENOENT handler"` cannot match today.
+- **Where:** `src/storage/fts-store.ts` (`escapeFts5`, `search`),
+  `src/retrieval/retriever.ts` (already boosts FTS for `code_lookup` intent).
+- **Verify:** the identifier-heavy query slice of R4 should improve first.
+
+**R3. Make compression query-aware.**
+- **Do:** add a `query` parameter to compression. Keep the fixed caps as the
+  fallback, but when a query is present, retain lines containing query tokens,
+  identifiers, imports and the signature, then truncate. Treat `full` as the
+  budget-generous level (evidence says budgets under ~8k tokens lose quality).
+- **Why:** SWE-Pruner shows task-conditioned pruning both cuts tokens and raises
+  success rates, while Finding 12 shows fixed aggressive caps hurt. Our current
+  caps can delete the only line that answers the question.
+- **Where:** `src/compression/compressor.ts` (`TRUNCATION_LIMITS`, prompts),
+  `src/retrieval/retriever.ts` (packing already estimates compressed size).
+- **Verify:** measure served tokens and answer quality per compression level on
+  the labeled set; expect equal or better recall at equal or fewer tokens.
+
+**R4. Build a labeled retrieval eval and report quality in the smoke run.**
+- **Do:** a small query set (start with 50-100 queries over this repo: bug-report
+  style, identifier style, NL style, file-name style) with expected file or
+  chunk labels. Report Recall@5, Recall@10, nDCG@10, a distraction ratio
+  (relevant chunks among served), tokens and latency. Add it to
+  `npm run smoke:real-world`.
+- **Why:** every other recommendation depends on being able to tell "better"
+  from "feels better". CORE-Bench shows code-search scores overstate agentic
+  usefulness, and UDCG shows distraction must be counted, not ignored. This also
+  gives our 97% savings claim a paired quality number.
+- **Where:** `scripts/smoke-real-world.mjs`, new `tests/eval/` fixture,
+  optionally a CORE-Bench sample from Hugging Face for external validity.
+- **Verify:** the harness itself: it must fail loudly when scores drop.
+
+### P1: next iteration
+
+**R5. Multi-path retrieval with a structured symbol path.**
+- **Do:** keep vector + FTS, and add a third path: exact lookup in the graph node
+  index for symbols and file paths named in the query. Fuse all three with the
+  existing RRF.
+- **Why:** CodeRAG attributes its gains to multi-path retrieval over a single
+  similarity path; the coding-agent study found real agents mix lexical,
+  semantic and structural signals. We extract identifiers today but only use
+  them for keyword distance.
+- **Where:** `src/retrieval/retriever.ts`, `src/retrieval/query-parser.ts`,
+  `src/storage/graph-store.ts`.
+- **Verify:** identifier and "where is X defined" query slices of R4.
+
+**R6. Small-to-big serving: expand hits with parent and neighbor context.**
+- **Do:** after ranking, expand the top hits with the enclosing module summary
+  and adjacent chunks in the same file, plus the file-level module chunk, until
+  the marginal-ratio stop. Keep `expand_chunk` for deeper drilling.
+- **Why:** cross-file context length was the dominant quality parameter in the
+  controlled study (+4.2 points from 2048 → 8192 tokens), and small-to-big is
+  the standard fix for "the chunk that retrieves well is not the chunk that
+  answers well". CORE-Bench also flags long, diluted chunks as a problem, so
+  expand selectively rather than making every chunk bigger.
+- **Where:** `src/retrieval/retriever.ts` (diversity/packing stage),
+  `src/indexer/chunker.ts` (emit module and top-level chunks for AST files too).
+- **Verify:** recall should rise while token count stays bounded by the budget.
+
+**R7. Make the graph a real retrieval path (context packs).**
+- **Do:** for each top hit, build a small context pack: the node, its file, its
+  callers/callees or importers/importees with edge types labeled, and a
+  centrality-based prior for which files are "important" in this repo.
+- **Why:** RepoGraph gained +2.66/+2.34 absolute on SWE-bench-Lite across four
+  systems at no extra cost; Aider's PageRank file ranking was observed working
+  in practice. Our current expansion is 2 chunks from 2 files with a magic 0.85.
+- **Where:** `src/storage/local-backend.ts` (`getRelatedFilePaths`),
+  `src/storage/graph-store.ts`, `src/retrieval/retriever.ts`,
+  `related_context` tool in `src/extension.ts`.
+- **Verify:** cross-file queries in R4; count how often the labeled answer file
+  arrives via graph edges instead of similarity.
+
+**R8. Adaptive retrieval: score the query, tell the agent, then optionally elaborate.**
+- **Do:** treat the score distribution (top score, margin to the second cluster,
+  agreement between vector and FTS) as a query-performance signal. When it is
+  low, return it in the tool result: "low confidence, try X". Only when the
+  agent opts in (or the query is classified ambiguous), generate an elaborated
+  variant and run both, keeping the raw query's results.
+- **Why:** predicted query performance correlates with answer quality in
+  agentic RAG, and Findings 1/2/4 say raw queries win by default while
+  elaboration helps only ambiguous ones. This gives us the loop real agents use
+  (broad first, narrow second) without semantic drift.
+- **Where:** `src/retrieval/confidence.ts`, `src/retrieval/query-parser.ts`,
+  `src/guidance.ts` (document the new signal for the agent).
+- **Verify:** measure how often a low-confidence signal is followed by a
+  successful second query in real sessions; track recall on ambiguous queries.
+
+**R9. Return refine suggestions with every search.**
+- **Do:** in `context_search` results, add 2-3 concrete next queries (sibling
+  symbol names, the file that almost matched, an alternate term seen in top
+  hits). Keep it to a line or two to stay token-cheap.
+- **Why:** every production coding agent in the exploratory study narrowed
+  iteratively, and RepoCoder's loop is the mechanism behind its reported gain.
+  Suggestions turn our one-shot tool into a guided loop at near-zero cost.
+- **Where:** tool response shaping in `src/extension.ts` or `engine.ts`.
+- **Verify:** follow-up query rate and whether sessions stop earlier.
+
+### P2: experiments, gated behind config, only with R4 in place
+
+**R10. Opt-in reranker.**
+- **Do:** `retrieval.rerank: off | cheap` using a small ONNX cross-encoder via
+  the transformers.js runtime we already ship (text-classification task).
+  Default `off`.
+- **Why:** evidence is genuinely split. The SE study found reranking harmful
+  and expensive (815 ms GPU), while CodeRAG and Anthropic found large gains on
+  repository tasks. Only our own eval can settle which side applies to pi's
+  workloads.
+- **Where:** new `src/retrieval/reranker.ts`, config in `src/config.ts`.
+- **Verify:** R4 with rerank on/off; enable by default only if it wins on
+  quality *and* the latency budget holds (warm queries are ~11 ms today).
+
+**R11. Optional code-specialized embedder.**
+- **Do:** document an `embedding.model: code` option (a code-oriented ONNX
+  model) next to the general default, and evaluate it before recommending it.
+- **Why:** Findings 7/8 say scale and code-specific training did not reliably
+  win, but CORE-Bench shows agentic retrieval is a different, harder regime
+  where in-domain training helped a lot. So: offer it, measure it, then decide.
+- **Where:** `src/indexer/embedder.ts` (`DEFAULT_LOCAL_MODEL`), README config block.
+- **Verify:** R4 per model; the local backend already reindexes on dimension change.
+
+**R12. Rework chunking toward density: split-merge with a budget.**
+- **Do:** adopt cAST's algorithm: top-down fit of AST nodes to a non-whitespace
+  character budget, greedy merge of small siblings, and emit module/top-level
+  chunks so code outside functions is retrievable. Keep chunks concatenable back
+  to the original file.
+- **Why:** function-only chunking was the worst measured strategy (−3.57 to
+  −5.64 points) and was never Pareto-optimal; cAST gains 4.3 Recall@5. Our
+  current chunker does exactly this (whole function/class nodes, no budget).
+- **Where:** `src/indexer/chunker.ts` (`walk`, `nodeToChunk`), chunk-id scheme in
+  `makeChunk` (ids include line spans, so the upstream column-offset fix must
+  survive the rework).
+- **Verify:** R4 by query type, plus re-run `smoke:real-world` for chunk counts
+  and token deltas; expect more, denser chunks.
+
+**R13. Link memories (A-MEM style).**
+- **Do:** when writing a decision or code area, compute links to related past
+  entries (embedding similarity + the `decided`/`discussed_in` graph edges) and
+  allow a later entry to annotate an earlier one instead of only appending.
+- **Why:** A-MEM's linked, evolving memories beat fixed-structure memory on six
+  models. Our memory is flat today, so recall quality degrades as history grows.
+- **Where:** `src/memory/db.ts`, `decision-extractor.ts`, `session-capture.ts`.
+- **Verify:** a small recall test: seed N decisions, query for each by topic,
+  measure whether its linked neighbours surface too.
+
+---
+
+## 4. What the literature says not to do
+
+- **No always-on LLM query rewriting**: semantic drift risk plus ~987 tokens per
+  rewrite (Finding 4 and the cost table).
+- **No default cross-encoder reranking**: often negative on SE tasks, expensive,
+  and our warm-query latency would regress (Findings 10-11).
+- **No aggressive compression**: fixed tight caps and hard truncation lose the
+  answering lines (Finding 12, SWE-Pruner).
+- **No chunk overlap by default**: two evaluations found no clear benefit.
+- **No "bigger embedder" reflex**: scale and code-specific training did not
+  decide outcomes (Findings 7/8). Measure first with R4.
+- **No full-file indexing as a substitute for retrieval**: useful for repos
+  under ~200k tokens, per Anthropic, which is a niche case.
+
+---
+
+## 5. Measurement plan
+
+1. **Baseline first**: run R4's harness on the current release and record
+   Recall@5/@10, nDCG@10, distraction ratio, tokens, latency.
+2. **One change at a time**: each Rn lands with its before/after numbers in the
+   PR. This mirrors how every paper above isolates one component.
+3. **Report quality in `smoke:real-world`**, next to the existing savings and
+   latency numbers, so regressions are visible without a separate tool.
+4. **External check**: optionally run a CORE-Bench level-2 slice to confirm
+   gains transfer beyond our own query set.
+5. **Session telemetry (opt-in)**: track low-confidence rate and follow-up
+   query rate to validate R8/R9 outside the lab.
+
+---
+
+## 6. Sources
+
+Primary sources read for this note (Evidence A unless marked):
+
+- **Not All RAGs Are Created Equal: A Component-Wise Empirical Study for SE Tasks**
+  (2026) https://arxiv.org/abs/2605.14503
+  Retriever beats generator; BM25 robust; query rewriting and reranking are
+  situational, not defaults.
+- **How Does Chunking Affect Retrieval-Augmented Code Completion?** (2026)
+  https://arxiv.org/abs/2605.04763
+  Function chunking worst; cross-file context length dominant.
+- **cAST: Enhancing Code RAG with Structural Chunking via AST** (2025)
+  https://arxiv.org/abs/2506.15655
+  Split-merge AST chunking, char budget, metadata retention.
+- **RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph**
+  (ICLR 2025) https://arxiv.org/abs/2410.14684
+  Graph plug-in: +2.66/+2.34 absolute SWE-bench-Lite resolve.
+- **CodeRAG: Finding Relevant and Necessary Knowledge for RAG Repository-Level
+  Code Completion** (EMNLP 2025) https://arxiv.org/abs/2509.16112
+  Log-probability query construction, multi-path retrieval, preference-aligned reranking.
+- **RepoCoder: Repository-Level Code Completion through Iterative Retrieval and
+  Generation** (EMNLP 2023) https://arxiv.org/abs/2303.12570
+  Iterative retrieve-generate loop for repo context.
+- **CORE-Bench: Code Retrieval in the Era of Agentic Coding** (2026)
+  https://arxiv.org/abs/2606.11864
+  Agentic retrieval is requirement-driven repo search; standard benchmarks overstate.
+- **SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents** (2026)
+  https://arxiv.org/abs/2601.16746
+  Task-conditioned pruning: 23-54% fewer tokens, success rates up.
+- **Provence: Efficient and Robust Context Pruning for RAG** (ICLR 2025)
+  https://arxiv.org/abs/2501.16214
+  Lightweight trained pruner removes irrelevant sentences pre-generation.
+- **LLMLingua: Compressing Prompts for Accelerated Inference** (EMNLP 2023)
+  https://arxiv.org/abs/2310.05736
+  Budget-controlled compression, up to 20x with little loss.
+- **Late Chunking: Contextual Chunk Embeddings** (2024)
+  https://arxiv.org/abs/2409.04701
+  +2.7-3.6% relative nDCG; overlap shows no clear benefit.
+- **Anthropic, Contextual Retrieval** (industry, engineering blog, 2024)
+  https://www.anthropic.com/engineering/contextual-retrieval
+  Failed retrievals −49% (embeddings + BM25), −67% with reranking.
+- **Lost in the Middle: How Language Models Use Long Contexts** (TACL 2024)
+  https://arxiv.org/abs/2307.03172
+  Result ordering matters: information in the middle of long context is used least.
+- **A-MEM: Agentic Memory for LLM Agents** (NeurIPS 2025)
+  https://arxiv.org/abs/2502.12110
+  Zettelkasten linking plus memory evolution beats fixed memory structures.
+- **MemGPT: Towards LLMs as Operating Systems** (2023)
+  https://arxiv.org/abs/2310.08560
+  Tiered, OS-style memory management for agents.
+- **Am I on the Right Track? Predicted Query Performance in Agentic RAG** (2025)
+  https://arxiv.org/abs/2507.10411
+  Query quality prediction correlates with answer quality: basis for adaptive retrieval.
+- **Search-R1: Training LLMs to Reason and Leverage Search Engines with RL** (2025)
+  https://arxiv.org/abs/2503.09516
+  Reference design for search-as-a-tool inside reasoning loops.
+- **Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE)** (ACL 2023)
+  https://arxiv.org/abs/2212.10496
+  Query→hypothetical answer→embedding; costs a generation per query.
+- **Redefining Retrieval Evaluation in the Era of LLMs** (2025)
+  https://arxiv.org/abs/2510.21440
+  UDCG metric counts distraction, +36% correlation with end-to-end accuracy.
+- **Retrieval-Augmented Code Generation: A Survey (Repository-Level)** (2025)
+  https://arxiv.org/abs/2510.04905
+  Taxonomy of repo-level retrieval: substrate, control regime, evaluation.
+- **Revisiting Code Search in a Two-Stage Paradigm** (2022) Evidence B
+  https://arxiv.org/abs/2208.11274
+  Retrieve-then-rerank for code search.
+- **CORNSTACK: High-Quality Contrastive Data for Code Retrieval and Reranking**
+  (ICLR 2025) Evidence B https://arxiv.org/abs/2412.01007
+  Rerankers matter for code retrieval when trained on good data.
+- **COIR: A Comprehensive Benchmark for Code Information Retrieval** (ACL 2025)
+  Evidence B https://aclanthology.org/2025.acl-long.1072.pdf
+  Broader code-IR benchmark beyond CodeSearchNet.
+- **CodeSearchNet Challenge** (2019) Evidence B https://arxiv.org/abs/1909.09436
+  The classic benchmark, and its limits.
+- **Voyage AI, How do we evaluate vector-based code retrieval?** (industry, 2024)
+  https://blog.voyageai.com/2024/12/04/code-retrieval-eval/
+  CoSQA ~51% bad labels; CodeSearchNet queries trivial; mine evals from SWE work.
+- **An Exploratory Study of Code Retrieval Techniques in Coding Agents** (2025
+  preprint, not peer reviewed) Evidence B
+  https://doi.org/10.20944/preprints202510.0924
+  Seven production agents all narrow iteratively and mix lexical/semantic/graph/LSP.
+- **ColBERT: Efficient Passage Search via Late Interaction** (SIGIR 2020)
+  Evidence B https://arxiv.org/abs/2004.12832
+  Late interaction: better than bi-encoders, heavier infrastructure than ours.
+
+---
+
+## 7. Open questions
+
+- Does the function-chunking penalty from the completion study transfer to our
+  question-answering workload? R4 + R12 will answer this empirically.
+- Which side of the reranking debate applies to pi: SE tasks or repo tasks?
+- Is a code-specialized embedder worth the extra model weight for local users,
+  given Findings 7/8?
+- Should confidence scoring learn weights (vector/keyword/recency) from the
+  labeled set instead of the fixed 0.5/0.4/0.1 split?

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pi-context-engine",
-	"version": "0.1.3",
+	"version": "0.1.4",
 	"description": "Semantic code context for pi: tree-sitter index + hybrid vector/BM25 retrieval + cross-session memory, all local. Port of code-context-engine (MIT, elara-labs) as a native pi extension.",
 	"keywords": [
 		"pi-package",
@@ -51,6 +51,9 @@
 		"web-tree-sitter": "^0.22.6",
 		"yaml": "^2.9.1"
 	},
+	"optionalDependencies": {
+		"@huggingface/transformers": "^4.3.1"
+	},
 	"peerDependencies": {
 		"@earendil-works/pi-ai": "*",
 		"@earendil-works/pi-coding-agent": "*",
@@ -60,7 +63,6 @@
 		"@earendil-works/pi-ai": ">=0.80.0",
 		"@earendil-works/pi-coding-agent": ">=0.99.2",
 		"@types/node": "^22.0.0",
-		"@huggingface/transformers": "^4.3.1",
 		"typebox": "^1.0.0",
 		"typescript": "^5.9.0",
 		"vitest": "^3.1.0"

package/src/indexer/embedder.ts

@@ -40,13 +40,37 @@ interface TransformersModule {
 	env: Record<string, unknown>;
 }
 
+export type TransformersLoader = () => Promise<TransformersModule>;
+
+const defaultLoadTransformers: TransformersLoader = async () =>
+	(await import("@huggingface/transformers")) as unknown as TransformersModule;
+
+/**
+ * The package ships `@huggingface/transformers` as an optional dependency, so a
+ * plain npm install has it, but installs done with `--omit=optional` do not.
+ * Detect that case so we can tell the user exactly how to fix it instead of
+ * crashing with a bare "Cannot find module".
+ */
+function isModuleNotFoundError(err: unknown): boolean {
+	const code = (err as NodeJS.ErrnoException | undefined)?.code;
+	if (code === "ERR_MODULE_NOT_FOUND" || code === "MODULE_NOT_FOUND") return true;
+	const message = (err as Error | undefined)?.message ?? "";
+	return message.includes("Cannot find module") || message.includes("Cannot find package");
+}
+
+const LOCAL_BACKEND_INSTALL_HINT =
+	"Local embedding backend unavailable: @huggingface/transformers is not installed. " +
+	"It ships as an optional dependency, so a plain install includes it. " +
+	"Fix: reinstall pi-context-engine (pi update npm:pi-context-engine), " +
+	"or force the Ollama backend with PCE_EMBED_BACKEND=ollama.";
+
 export class LocalBackend implements EmbeddingBackend {
 	readonly name = "local";
 	readonly modelName: string;
 	dimension = 0;
 	private pipelinePromise: Promise<(inputs: string | string[]) => Promise<number[][]>> | null = null;
 
-	constructor(modelName: string = DEFAULT_LOCAL_MODEL) {
+	constructor(modelName: string = DEFAULT_LOCAL_MODEL, private readonly loadTransformers: TransformersLoader = defaultLoadTransformers) {
 		this.modelName = modelName;
 	}
 
@@ -55,7 +79,16 @@ export class LocalBackend implements EmbeddingBackend {
 			this.pipelinePromise = (async () => {
 				const cacheDir = resolveModelCacheDir();
 				fs.mkdirSync(cacheDir, { recursive: true });
-				const transformers = (await import("@huggingface/transformers")) as unknown as TransformersModule;
+				let transformers: TransformersModule;
+				try {
+					transformers = await this.loadTransformers();
+				} catch (err) {
+					// Drop the memoized failure so a retry (after installing the
+					// package) gets a fresh chance instead of replaying the error.
+					this.pipelinePromise = null;
+					if (isModuleNotFoundError(err)) throw new Error(LOCAL_BACKEND_INSTALL_HINT);
+					throw err;
+				}
 				transformers.env.cacheDir = cacheDir;
 				const extractor = await transformers.pipeline("feature-extraction", this.modelName, {
 					pooling: "mean",