@alextis59/athena 1.1.0 → 1.3.0

package/docs/implementation-plan.md

@@ -1,1433 +0,0 @@
-# Athena Implementation Plan
-
-Date: 2026-07-03
-
-Related documents:
-
-- [Agentic RAG Tool Design Study](./agentic-rag-study.md)
-- [Agentic RAG Tool Specifications](./agentic-rag-specifications.md)
-
-## Purpose
-
-This document turns the study and specifications into a step-by-step execution
-plan for building `athena`, the local-first documentation RAG command.
-
-It is also the project progress ledger. Update the checkboxes as implementation
-work lands, and only mark an item complete when the matching proof command,
-test, or manual validation has passed.
-
-## Ground Rules
-
-- The project name is `athena`.
-- The npm package name is `@alextis59/athena`.
-- The CLI binary name is `athena`.
-- `npx @alextis59/athena` is only the package-run invocation.
-- Runtime cache lives in `.athena/`.
-- Athena owns the agentic loop. No agent SDK should control tool selection,
-  tool execution, step limits, memory, streaming, or citation validation.
-- `pdf-2-llm` is the canonical PDF ingestion dependency.
-- Local operation is the default. Remote model or embedding providers require
-  explicit user configuration.
-- Citations are product data, not display-only strings.
-- MCP and external coding-agent support are integrations, not the core runtime.
-
-## Progress Snapshot
-
-- [x] Study document committed.
-- [x] Specifications document committed.
-- [x] Implementation plan created.
-- [x] Validation document corpus retrieved and locked.
-- [x] Code scaffold created.
-- [x] First runnable `athena` command available.
-- [x] Markdown-only local RAG MVP complete.
-- [x] Local embeddings and hybrid search complete.
-- [x] PDF RAG with `pdf-2-llm` complete.
-- [x] Evaluation harness complete.
-- [x] MCP bridge complete.
-
-## Completion Policy
-
-For this plan, "done" means all of the following are true:
-
-- the implementation exists in the repository;
-- automated tests or documented manual checks pass;
-- behavior matches the specifications;
-- citations remain source-backed;
-- the work is committed;
-- this file has been updated with checked items and proof notes.
-
-Do not check off implementation items for partial scaffolding unless the item
-explicitly says it is scaffolding.
-
-## Milestone 0: Repository Foundation
-
-Goal: establish a maintainable Node/TypeScript project with a real `athena`
-binary, repeatable local checks, and a clear module layout.
-
-### 0.1 Package and Tooling
-
-- [x] Decide the package manager and lockfile policy.
-  Proof: repository contains one committed lockfile or an explicit decision
-  note explaining why no lockfile is committed yet.
-- [x] Create `package.json` with package name, `bin.athena`, module type, and
-  public/private package decision.
-  Proof: `npm pkg get name bin` shows the expected package and binary fields.
-- [x] Add TypeScript configuration.
-  Proof: `npm run typecheck` runs and reports no compiler errors.
-- [x] Add source formatting and linting commands.
-  Proof: `npm run lint` and formatting checks run locally.
-- [x] Add test runner setup.
-  Proof: `npm test` runs at least one smoke test.
-- [x] Add build script that emits runnable CLI output.
-  Proof: `npm run build` completes and creates the expected distribution files.
-- [x] Add `.gitignore` entries for local build artifacts, model caches, and
-  `.athena/` runtime indexes.
-  Proof: `git status --short` stays clean after running build and basic index
-  commands.
-
-### 0.2 Project Layout
-
-- [x] Create the baseline `src/` layout:
-  `cli/`, `config/`, `storage/`, `indexer/`, `parsers/`, `retrieval/`,
-  `embeddings/`, `agent/`, `server/`, `ui/`, and `mcp/`.
-  Proof: source folders exist and are referenced by the build.
-- [x] Add stable shared types for sources, documents, chunks, locators,
-  citations, parser artifacts, tool results, and provider events.
-  Proof: typecheck exercises imports across at least two modules.
-- [x] Add central error types for user-facing CLI failures and internal failures.
-  Proof: CLI smoke tests can assert exit code and message shape.
-- [x] Add internal logging helpers with quiet/default/debug modes.
-  Proof: a test or smoke command verifies log level filtering.
-
-### 0.3 CLI Shell
-
-- [x] Implement the top-level `athena` CLI entrypoint.
-  Proof: `node dist/cli/main.js --help` or equivalent displays Athena help.
-- [x] Add subcommands: default, `index`, `serve`, `ask`, `doctor`, and `mcp`.
-  Proof: each subcommand appears in CLI help.
-- [x] Add common options: `--root`, `--config`, `--db`, `--host`, `--port`,
-  `--debug`, and `--json` where applicable.
-  Proof: CLI parser tests cover option parsing and default values.
-- [x] Add process exit-code policy for user errors, runtime failures, and
-  interrupted commands.
-  Proof: tests cover at least one success and one failure path.
-
-Milestone 0 proof, 2026-07-03:
-
-- `npm pkg get name bin type main types scripts` reported package name
-  `@alextis59/athena`, binary `./dist/cli/main.js`, ESM module type,
-  declaration entry, and build/lint/test/typecheck scripts.
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 6 smoke tests.
-- `npm run build` passed.
-- `node dist/cli/main.js --help` displayed Athena help with `default`,
-  `index`, `serve`, `ask`, `doctor`, and `mcp`.
-
-## Milestone 1: Configuration and Storage Core
-
-Goal: load optional configuration, create `.athena/index.sqlite`, and maintain
-versioned database state before indexing real documents.
-
-### 1.1 Configuration
-
-- [x] Implement config discovery for `athena.config.ts`,
-  `athena.config.mjs`, and `athena.config.json`.
-  Proof: tests load each supported config type from a fixture.
-- [x] Add `defineConfig()` for typed user configuration.
-  Proof: TypeScript fixture imports `defineConfig` from the package.
-- [x] Implement default config values matching the specifications.
-  Proof: snapshot or object tests verify include/exclude patterns, server
-  defaults, local embedding defaults, and PDF defaults.
-- [x] Validate user config with clear errors.
-  Proof: invalid config fixture fails with field-level diagnostics.
-- [x] Add remote-provider opt-in checks for embeddings and chat models.
-  Proof: tests verify document text cannot be sent remotely unless explicitly
-  configured.
-
-### 1.2 Database Foundation
-
-- [x] Select and document the SQLite driver.
-  Proof: a short decision note is added near storage code or in a committed
-  ADR-style comment/doc.
-- [x] Create `.athena/` directory management.
-  Proof: `athena doctor --root <fixture>` reports missing and initialized cache
-  states clearly.
-- [x] Add migration runner with schema version tracking.
-  Proof: migration tests create a fresh database and upgrade an older fixture.
-- [x] Create required tables: `sources`, `documents`, `chunks`, `chunks_fts`,
-  `embeddings`, `runs`, `parser_artifacts`, `conversation_turns`, and
-  `answer_citations`.
-  Proof: schema test verifies all required tables and key columns.
-- [x] Add indexes for source paths, hashes, chunk source IDs, embedding cache
-  keys, and citation lookups.
-  Proof: schema test checks required indexes.
-- [x] Add transaction helper for index updates.
-  Proof: rollback test verifies partial index writes do not persist.
-
-### 1.3 Doctor Command
-
-- [x] Implement `athena doctor` database checks.
-  Proof: doctor output includes database status, schema version, and table
-  availability.
-- [x] Add parser availability checks.
-  Proof: doctor reports `pdf-2-llm` availability without importing it during
-  non-PDF commands unnecessarily.
-- [x] Add embedding provider diagnostics.
-  Proof: doctor reports local model cache state and configured provider.
-- [x] Add UI port availability diagnostics.
-  Proof: test or manual check covers occupied and free port cases.
-
-Milestone 1 proof, 2026-07-03:
-
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 16 tests covering config formats, typed `defineConfig`,
-  validation diagnostics, remote-provider opt-in, migrations, schema tables and
-  indexes, transaction rollback, doctor database states, and port diagnostics.
-- `npm run build` passed.
-- Built CLI check: `node dist/cli/main.js doctor --root <tmp> --port 4318`
-  reported a missing cache with schema unavailable.
-- Built storage and doctor check: initializing the same temp root through
-  `initializeDatabase()` then running
-  `node dist/cli/main.js doctor --root <tmp> --port 4318 --json` reported
-  schema version 1 and all required tables/indexes present.
-
-## Milestone 2: Validation Document Corpus Retrieval
-
-Goal: collect a representative set of real and synthetic documents early, so
-parser, indexing, retrieval, citation, PDF, and UI behavior are validated
-against varied source material instead of only hand-made unit fixtures.
-
-### 2.1 Corpus Policy and Manifest
-
-- [x] Define where validation documents live.
-  Proof: a committed corpus note documents committed fixtures, downloaded
-  artifacts, ignored cache paths, and expected directory layout.
-- [x] Define licensing and redistribution rules.
-  Proof: the corpus note distinguishes committed license-compatible samples
-  from download-only third-party documents.
-- [x] Create a corpus manifest schema.
-  Proof: manifest entries include ID, document type, source URL or generator,
-  license, retrieval date, expected hash, size, language, and intended test
-  coverage.
-- [x] Decide how large third-party files are cached.
-  Proof: the plan documents whether large files live under an ignored local
-  cache such as `.athena/corpus-cache/` or a dedicated ignored fixture cache.
-- [x] Add checksum verification requirements for downloaded samples.
-  Proof: the retrieval plan rejects changed files unless the manifest is
-  intentionally updated.
-
-### 2.2 Real Document Retrieval Targets
-
-- [x] Retrieve Markdown documentation samples.
-  Proof: corpus manifest includes at least a README-style guide, an API
-  reference page, and a changelog or migration note.
-- [x] Retrieve MDX documentation samples.
-  Proof: corpus includes MDX with headings, fenced code, imports, and embedded
-  JSX that should not be executed.
-- [x] Retrieve plain text and extensionless documentation samples.
-  Proof: corpus includes files such as LICENSE, CHANGELOG, NOTICE, or CLI help
-  text with stable line ranges.
-- [x] Retrieve reStructuredText documentation samples.
-  Proof: corpus includes RST headings, directives, lists, and code blocks.
-- [x] Retrieve AsciiDoc documentation samples.
-  Proof: corpus includes AsciiDoc sections, lists, tables, and code blocks.
-- [x] Retrieve HTML documentation samples.
-  Proof: corpus includes static HTML with navigation, scripts/styles that must
-  be ignored, headings, links, and code examples.
-- [x] Retrieve born-digital PDF samples.
-  Proof: corpus includes searchable PDFs with paragraphs, headings, page
-  numbers, and extractable text.
-- [x] Retrieve table-heavy PDF samples.
-  Proof: corpus includes PDFs where table sidecars and page-region citations
-  are necessary to answer validation questions.
-- [x] Retrieve hard PDF samples.
-  Proof: corpus includes at least one malformed, encrypted, scanned,
-  low-confidence, or unsupported PDF case with expected warning behavior.
-- [x] Retrieve mixed-language or non-English samples when practical.
-  Proof: corpus manifest records language and script coverage for any
-  multilingual, RTL, or CJK samples.
-
-### 2.3 Synthetic Control Documents
-
-- [x] Add synthetic documents with known answers and exact citations.
-  Proof: fixtures contain deterministic questions with expected path, line, or
-  page citations.
-- [x] Add ambiguous-heading controls.
-  Proof: corpus contains repeated section titles across different files so
-  retrieval must use path and surrounding context.
-- [x] Add exact-symbol controls.
-  Proof: corpus contains API names, CLI flags, environment variables, and
-  config keys that must rank above vague conceptual matches.
-- [x] Add semantic-only controls.
-  Proof: corpus contains answerable questions where the relevant text does not
-  share exact query terms.
-- [x] Add stale or superseded-document controls.
-  Proof: corpus includes old and current versions so ranking can penalize
-  superseded chunks.
-
-### 2.4 Retrieval Script and Ground Truth
-
-- [x] Add a corpus retrieval command or script.
-  Proof: one command downloads or generates the corpus into the documented
-  location without modifying source implementation files.
-- [x] Add offline fallback behavior.
-  Proof: the command can verify an already-cached corpus without network
-  access.
-- [x] Add ground-truth question files.
-  Proof: every required document type has at least one question with expected
-  answer facts and citation targets.
-- [x] Add negative questions.
-  Proof: ground truth includes questions that should be refused or marked
-  unsupported because the answer is absent from the corpus.
-- [x] Add corpus health reporting.
-  Proof: the retrieval command reports document counts by type, hash status,
-  missing files, and license/redistribution status.
-
-### 2.5 Corpus Acceptance
-
-- [x] Corpus covers every first-class document type.
-  Proof: health output reports Markdown, MDX, text, extensionless, RST,
-  AsciiDoc, HTML, and PDF samples.
-- [x] Corpus includes realistic real-world documents.
-  Proof: manifest contains source URLs, retrieval dates, and checksums for
-  downloaded documents.
-- [x] Corpus includes deterministic synthetic controls.
-  Proof: committed fixtures and ground-truth files make parser and retrieval
-  failures reproducible.
-- [x] Corpus can be refreshed intentionally.
-  Proof: manifest update workflow records changed hashes and review notes.
-- [x] Corpus can validate the MVP before embeddings exist.
-  Proof: at least one Markdown/text/RST/AsciiDoc/HTML question set is answerable
-  through lexical retrieval only.
-- [x] Corpus can validate PDF behavior once `pdf-2-llm` is integrated.
-  Proof: PDF question set requires page, region, confidence, warning, and table
-  sidecar metadata where applicable.
-
-Milestone 2 proof, 2026-07-03:
-
-- `test/corpus/manifest.schema.json` defines the manifest schema.
-- `docs/validation-corpus.md` documents committed fixtures, ignored download
-  cache paths, licensing, redistribution policy, checksum policy, and refresh
-  workflow.
-- `npm run corpus:retrieve` downloaded or verified 19 entries: Markdown,
-  MDX, text, extensionless, RST, AsciiDoc, HTML, PDF, and French HTML
-  non-English coverage.
-- `npm run corpus:health` verified the already-cached corpus offline with
-  `Hash status: ok=19`.
-- `npm test` passed manifest and ground-truth coverage tests, including
-  negative-question coverage and every first-class document type.
-
-## Milestone 3: File Discovery and Incremental Indexing
-
-Goal: discover supported docs, skip unchanged files, and record source metadata
-deterministically.
-
-### 3.1 Discovery
-
-- [x] Implement file discovery from the configured root.
-  Proof: fixture test returns Markdown, MDX, text, RST, AsciiDoc, HTML, PDF, and
-  extensionless documentation files.
-- [x] Apply default exclusions: `.git/**`, `node_modules/**`, `.athena/**`,
-  `dist/**`, `build/**`, `coverage/**`, and unsupported binary files.
-  Proof: discovery fixture confirms excluded paths are omitted.
-- [x] Add include/exclude override support from config.
-  Proof: tests cover custom include and exclude patterns.
-- [x] Normalize paths to stable repository-relative paths.
-  Proof: tests pass on paths with nested directories and spaces.
-- [x] Classify file kind before parsing.
-  Proof: tests map extensions and extensionless names to expected kinds.
-
-### 3.2 Hashing and Source State
-
-- [x] Hash source bytes for every discovered file.
-  Proof: tests verify changed contents alter the hash.
-- [x] Store path, kind, size, mtime, hash, parser name, parser version, parser
-  options hash, and indexed timestamp.
-  Proof: database tests inspect inserted `sources` rows.
-- [x] Skip unchanged files on repeat indexing.
-  Proof: `athena index` run twice against a fixture reports zero changed files
-  on the second run.
-- [x] Reindex changed files and delete stale chunks.
-  Proof: modifying one fixture file only replaces that file's document and
-  chunk rows.
-- [x] Remove index rows for deleted files.
-  Proof: deleting a fixture source removes its source, document, chunks,
-  embeddings, parser artifacts, and FTS rows.
-
-### 3.3 Index Command
-
-- [x] Implement `athena index`.
-  Proof: command exits zero after indexing a Markdown fixture.
-- [x] Add progress output for discovered, skipped, parsed, embedded, failed, and
-  committed counts.
-  Proof: CLI snapshot or manual output includes all count fields.
-- [x] Add `--json` output mode for automation.
-  Proof: command emits parseable JSON with the same counts.
-- [x] Exit non-zero on parser, migration, or embedding failures.
-  Proof: fixture with forced parser failure returns a non-zero exit code.
-- [x] Persist run metadata in `runs`.
-  Proof: database test verifies run status and timestamps.
-
-Milestone 3 proof, 2026-07-03:
-
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 24 tests, including kind classification, default
-  exclusions, include/exclude overrides, hash changes, repeat indexing skips,
-  changed-file updates, deleted-file cleanup, non-zero database failure, and
-  persisted run metadata.
-- `npm run build` passed.
-- Built CLI check: indexing a temp root with `README.md` and `docs/guide.md`
-  returned JSON with `discovered=2`, `committed=2`, then a second run returned
-  `skipped=2`, and SQLite contained 2 sources plus 2 index runs.
-
-## Milestone 4: Text Document Parsing and Source-Aware Chunking
-
-Goal: convert non-PDF documentation into chunks with stable line and heading
-locators.
-
-### 4.1 Markdown and MDX
-
-- [x] Implement Markdown parser with line positions.
-  Proof: parser tests produce document text and heading path metadata.
-- [x] Preserve fenced code blocks as searchable text.
-  Proof: fixture query can find an identifier inside a code block.
-- [x] Preserve heading hierarchy for chunk metadata.
-  Proof: chunks include heading paths such as `Authentication > Tokens`.
-- [x] Support MDX files without executing embedded code.
-  Proof: MDX fixture parses safely and stores line ranges.
-- [x] Store parser warnings for malformed or partially parsed documents.
-  Proof: fixture with malformed Markdown records a warning without crashing.
-
-### 4.2 Plain Text and Markup Formats
-
-- [x] Implement `.txt` and extensionless documentation parser.
-  Proof: README and CHANGELOG-style fixtures produce line-located chunks.
-- [x] Implement `.rst` parser fallback.
-  Proof: RST fixture indexes headings and body text.
-- [x] Implement `.adoc` parser fallback.
-  Proof: AsciiDoc fixture indexes headings and body text.
-- [x] Implement HTML parser that extracts visible text and heading structure.
-  Proof: HTML fixture omits scripts/styles and includes visible headings.
-- [x] Store normalized text separately from source text when useful for search.
-  Proof: database row includes both text and normalized text fields.
-
-### 4.3 Chunking
-
-- [x] Implement deterministic chunking with target size, overlap, and hard
-  maximum controls.
-  Proof: chunking tests are stable across repeated runs.
-- [x] Keep chunks within source-aware boundaries when possible.
-  Proof: headings, lists, tables, and code blocks are not split unnecessarily in
-  fixtures.
-- [x] Store locators as structured JSON.
-  Proof: chunk rows include line ranges for text-like files.
-- [x] Estimate token counts per chunk.
-  Proof: chunk rows contain token estimates and tests cover long documents.
-- [x] Add neighboring chunk relationships by source and ordinal.
-  Proof: retrieval tests can fetch before/after chunks.
-
-Milestone 4 proof, 2026-07-03:
-
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 30 tests, including Markdown/MDX parsing, malformed
-  Markdown warnings, plain text, RST, AsciiDoc, HTML script/style omission,
-  deterministic chunking, indexed documents/chunks, structured line locators,
-  token estimates, and previous/next chunk IDs.
-- `npm run build` passed.
-- Built CLI check: indexing a temp Markdown root returned `parsed=1`,
-  `committed=1`, and SQLite contained one document plus a line-located chunk
-  with title `Auth`.
-
-## Milestone 5: Lexical Retrieval and Core Tools
-
-Goal: make indexed Markdown searchable and expose citation-aware tools before
-adding semantic search.
-
-### 5.1 SQLite FTS5
-
-- [x] Create and populate `chunks_fts`.
-  Proof: schema and indexing tests verify FTS rows for each searchable chunk.
-- [x] Implement lexical search with ranking and snippets.
-  Proof: exact symbol fixture returns the expected chunk first.
-- [x] Add path-prefix and kind filters.
-  Proof: retrieval tests restrict results by `docs/` and `markdown`.
-- [x] Boost heading and exact identifier matches.
-  Proof: ranking fixture places heading match above body-only match.
-- [x] Penalize stale or superseded chunks.
-  Proof: stale chunk fixture does not appear in normal retrieval results.
-
-### 5.2 Tool Runtime
-
-- [x] Define JSON schemas for `searchDocs`, `readSourceRange`, `grepDocs`,
-  `listDocs`, and `relatedChunks`.
-  Proof: tool schema tests validate accepted and rejected inputs.
-- [x] Implement `searchDocs` over FTS-only retrieval for this milestone.
-  Proof: tool test returns chunk IDs, paths, headings, snippets, scores, and
-  locators.
-- [x] Implement `readSourceRange`.
-  Proof: tool test reads exact line ranges from indexed source files.
-- [x] Implement `grepDocs` for exact string and bounded regex search.
-  Proof: tool test finds identifiers and rejects unsafe or unbounded patterns.
-- [x] Implement `listDocs`.
-  Proof: tool test lists indexed sources filtered by path prefix and kind.
-- [x] Implement `relatedChunks`.
-  Proof: tool test returns configured neighbors around a chunk ID.
-- [x] Track which chunks and source ranges are retrieved or read during a turn.
-  Proof: citation validation tests can distinguish seen and unseen sources.
-
-Milestone 5 proof, 2026-07-03:
-
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 33 tests, including FTS population, exact-symbol ranking,
-  path/kind filters, stale-result penalty, tool schemas, `searchDocs`,
-  `readSourceRange`, `grepDocs`, `listDocs`, `relatedChunks`, and per-turn
-  retrieval/read trace tracking.
-- `npm run build` passed.
-
-## Milestone 6: Local Server and Chat UI Shell
-
-Goal: launch a local UI that can inspect the index, search sources, and display
-citations before the full agent loop is enabled.
-
-### 6.1 Server
-
-- [x] Implement `athena serve`.
-  Proof: command starts an HTTP server bound to `127.0.0.1` by default.
-- [x] Add configurable host and port.
-  Proof: server tests or manual checks verify configured binding.
-- [x] Add index status endpoint.
-  Proof: endpoint returns schema version, source counts, chunk counts, and last
-  run status.
-- [x] Add retrieval endpoints backed by the core tools.
-  Proof: API tests search, read source ranges, and list docs.
-- [x] Add source preview endpoint with path validation.
-  Proof: tests reject path traversal and unknown files.
-- [x] Add graceful shutdown handling.
-  Proof: integration test or manual check confirms process exits cleanly.
-
-### 6.2 UI
-
-- [x] Create chat panel shell.
-  Proof: HTTP UI smoke test fetches the chat surface from the live local
-  server.
-- [x] Add index status indicator.
-  Proof: UI shows ready, stale, indexing, and error states from fixtures or
-  mocked responses.
-- [x] Add source citation list.
-  Proof: UI contract test includes citation actions and source metadata fields.
-- [x] Add source preview panel for text line ranges.
-  Proof: API test reads exact Markdown lines, and UI contract test includes the
-  preview panel and line-open action.
-- [x] Add retrieval trace drawer.
-  Proof: search API returns selected chunk trace data, and UI contract test
-  includes the trace panel.
-- [x] Add source filter controls.
-  Proof: API test verifies path and kind filtering, and UI contract test
-  includes the controls.
-- [x] Preserve copied answer citations.
-  Proof: copy action includes citation labels in copied text.
-
-Milestone 6 proof, 2026-07-03:
-
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm run typecheck` passed.
-- `npm test` passed 35 tests, including a live `startAthenaServer()` integration
-  test for `/`, `/api/status`, `/api/search`, `/api/read`, `/api/docs`,
-  `/api/grep`, `/api/related`, and `/api/source`.
-- `npm run build` passed.
-- Built-output smoke passed: `node dist/cli/main.js index --root <tmp> --json`
-  committed one Markdown source, `startAthenaServer()` from `dist/index.js`
-  served the UI, and `/api/search?q=ATHENA_INDEX_BATCH_SIZE` returned
-  `docs/guide.md`.
-
-Milestone 6 UI status and copy proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Focused
-  `npm test -- --test-name-pattern "server exposes status|deriveIndexStatusIndicator|renderAppHtml"`
-  passed 59 tests, including the `/api/status` ready indicator,
-  ready/stale/indexing/error status-state derivation, status indicator markup,
-  and copy behavior that sends snippet text plus a bracketed citation label to
-  the clipboard helper.
-
-## Milestone 7: Custom Agent Loop and Markdown RAG MVP
-
-Goal: answer questions from Markdown sources using Athena-owned tool calling and
-citation validation.
-
-### 7.1 Model Provider Interface
-
-- [x] Define `ModelProvider.streamResponse()`.
-  Proof: type tests or unit tests exercise streamed message, tool-call, and
-  completion events.
-- [x] Implement a first chat-generation provider adapter.
-  Proof: configured provider can stream a simple answer in an integration
-  check.
-- [x] Keep provider SDKs behind adapters only.
-  Proof: the agent loop imports Athena interfaces, not provider-owned agent
-  runtimes.
-- [x] Add model configuration and environment validation.
-  Proof: missing credentials or model settings fail with actionable messages.
-- [x] Add deterministic fake provider for tests.
-  Proof: agent loop tests run without network access.
-
-Milestone 7.1 proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Provider-focused `npm test -- --test-name-pattern
-  "ModelProvider|Ollama|FakeModelProvider|validateModelProviderConfig"` passed
-  38 tests, including scripted fake-provider events, model config validation,
-  and an Ollama adapter stream using a fake `fetch`.
-
-### 7.2 Loop Execution
-
-- [x] Build model messages from the current question, compact memory, and
-  retrieval policy.
-  Proof: tests inspect message construction for first and follow-up turns.
-- [x] Expose Athena tool schemas to the model.
-  Proof: test verifies available tools and schemas match implemented tools.
-- [x] Validate tool-call inputs before execution.
-  Proof: invalid path, range, limit, and regex inputs are rejected.
-- [x] Execute tool calls and append tool results.
-  Proof: loop test performs search then source read.
-- [x] Enforce default max step count of 6.
-  Proof: fake-provider runaway loop stops with a clear tool-budget message.
-- [x] Stream partial output to CLI and UI consumers.
-  Proof: stream test observes incremental events before completion.
-- [x] Persist compact conversation turns.
-  Proof: `conversation_turns` rows store user question, answer summary,
-  citations, timestamp, and optional pinned sources.
-
-Milestone 7.2 proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Loop-focused `npm test -- --test-name-pattern
-  "runAgentTurn|buildModelMessages|createModelToolDefinitions"` passed 43 tests,
-  including message construction, tool schema exposure, search/read tool
-  execution, invalid path rejection, step-budget stop behavior, streaming
-  message deltas, and `conversation_turns` persistence.
-
-### 7.3 Citation Validation
-
-- [x] Define normalized citation data model for text-like sources.
-  Proof: citation tests parse and serialize path, line range, chunk ID, and
-  label.
-- [x] Validate cited paths exist in the index.
-  Proof: unknown-path citation is rejected.
-- [x] Validate line ranges are inside file bounds.
-  Proof: out-of-range citation is rejected.
-- [x] Validate cited chunks or source ranges were retrieved or read during the
-  turn.
-  Proof: unseen-source citation is rejected.
-- [x] Require citations for substantive doc-derived claims.
-  Proof: answer test marks unsupported answer as refused or unsupported.
-- [x] Store validated answer citations in `answer_citations`.
-  Proof: database test verifies citation rows after a successful answer.
-
-Milestone 7.3 proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Citation-focused `npm test -- --test-name-pattern "citation|runAgentTurn"`
-  passed 48 tests, including text citation parse/serialize, unknown-path
-  rejection, out-of-range rejection, unread-source rejection, substantive
-  answer citation requirement, and `answer_citations` persistence.
-
-### 7.4 Markdown MVP Acceptance
-
-- [x] `athena index` indexes Markdown files.
-  Proof: fixture command creates sources, documents, chunks, and FTS rows.
-- [x] Unchanged Markdown files are skipped on the second run.
-  Proof: second index command reports skipped files and no rewritten chunks.
-- [x] `athena serve` starts a local UI.
-  Proof: HTTP UI smoke test opens the UI.
-- [x] A question can be answered from Markdown.
-  Proof: integration test asks a known fixture question and receives the
-  expected cited answer.
-- [x] Answer citations open exact line ranges.
-  Proof: API and agent tests open the cited source range.
-- [x] `athena ask "..."` returns terminal-readable citations.
-  Proof: CLI test checks answer text and citation label output.
-
-Milestone 7.4 proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Ask-focused `npm test -- --test-name-pattern
-  "runCli ask|terminal citations|requires a question"` passed 50 tests,
-  including `athena ask` with an injected deterministic provider, a Markdown
-  source-range read, streamed answer text, terminal `Citations:` output, and the
-  missing-question usage error.
-
-## Milestone 8: Local Embeddings and Hybrid Search
-
-Goal: improve retrieval quality with local embeddings, cached vectors, and
-hybrid ranking while preserving local-first privacy.
-
-### 8.1 Embedding Provider Interface
-
-- [x] Define embedding provider interface with model metadata.
-  Proof: unit tests cover provider name, model, dimensions, normalization, and
-  revision/digest fields.
-- [x] Implement Transformers.js or ONNX local provider.
-  Proof: local fixture generates embeddings without remote network calls after
-  model availability is satisfied.
-- [x] Set default model to `mixedbread-ai/mxbai-embed-xsmall-v1`.
-  Proof: default config test verifies provider, model, dimensions, pooling, and
-  normalization values.
-- [x] Add batching and progress reporting.
-  Proof: indexing output reports embedding batch progress for large fixtures.
-- [x] Add explicit opt-in provider adapters for Ollama, OpenAI, Voyage, Cohere,
-  and Gemini as separate follow-up tasks.
-  Proof: each remote adapter has config validation and privacy warnings before
-  document text can be sent.
-
-### 8.2 Vector Storage
-
-- [x] Select initial vector backend: `sqlite-vec` or MVP in-process scan.
-  Proof: decision is documented with migration implications.
-- [x] Store vector metadata in `embeddings`.
-  Proof: database test verifies chunk ID, cache key, provider metadata, and
-  dimensions.
-- [x] Compute embedding cache keys from chunk text hash, provider, model,
-  dimensions, pooling, normalization, quantization, revision/digest, and
-  chunking strategy version.
-  Proof: cache-key tests change each input and observe key changes.
-- [x] Skip unchanged vectors.
-  Proof: repeat indexing does not regenerate embeddings for unchanged chunks.
-- [x] Invalidate only affected vectors when embedding model settings change.
-  Proof: test changes model config and verifies source parsing is not repeated.
-
-### 8.3 Hybrid Retrieval
-
-- [x] Implement vector search.
-  Proof: semantic fixture returns `semantic-auth.md` for `login credentials`
-  without an FTS score.
-- [x] Merge FTS and vector results with deduplication.
-  Proof: hybrid test verifies unique chunk IDs and stable merged scores.
-- [x] Add neighboring context expansion.
-  Proof: related chunk expansion improves context returned to the agent.
-- [x] Add ranking boosts for identifiers, headings, source titles, and recently
-  read related chunks.
-  Proof: ranking tests cover each boost.
-- [x] Add ranking penalties for overly generic chunks.
-  Proof: generic overview fixture is not over-promoted.
-- [x] Display retrieval trace in the UI.
-  Proof: UI trace shows FTS score, vector score, merged score, and selected
-  context.
-
-### 8.4 Embeddings Acceptance
-
-- [x] Default local embeddings are generated.
-  Proof: `athena index` creates embeddings for fixture chunks.
-- [x] Vectors are cached.
-  Proof: repeat indexing skips unchanged vectors.
-- [x] Lexical and vector search are merged.
-  Proof: retrieval tests compare FTS-only, vector-only, and hybrid results.
-- [x] Changing the embedding model invalidates only affected vectors.
-  Proof: model-change test preserves source and parser rows while rebuilding
-  vector rows.
-
-Milestone 8 embedding storage proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Embedding-focused `npm test -- --test-name-pattern
-  "Embedding|embedding|athena index records|model settings change"` passed 54
-  tests, including deterministic local vector metadata, cache-key invalidation
-  inputs, vector encode/decode, default index-time embedding rows, repeat-index
-  vector cache hits, and model-change vector refresh without source reparsing.
-- Initial vector backend decision: store Float32 vectors as SQLite BLOBs and use
-  an MVP in-process scan for the upcoming hybrid retrieval step. This avoids a
-  migration dependency while the retrieval scorer is still evolving.
-
-Milestone 8 local Transformers.js provider proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "TransformersJsEmbeddingProvider|DeterministicEmbeddingProvider|stores
-  embedding vectors|embedding batch progress|model settings change"` passed 58
-  tests, including a fake local feature-extraction loader with
-  `local_files_only: true`, mean pooling, normalization, tensor-to-vector
-  conversion, deterministic fallback when local model files are unavailable,
-  and index-time provider preparation before cache-key calculation.
-- Built smoke passed: `node dist/cli/main.js index --root "$tmp" --json`
-  recorded `provider: "transformers-js"`, `dimensions: 384`, and
-  `implementation: "transformers-js-local-deterministic"` metadata when no
-  local model cache was present.
-
-Milestone 8 remote embedding adapter proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Provider-focused `npm test -- --test-name-pattern` run passed 92 tests,
-  including provider-specific REST payloads for Ollama, OpenAI, Voyage, Cohere,
-  and Gemini; injected fetch adapters with no live network calls; API-key
-  validation for cloud providers; explicit
-  `allowRemoteDocumentText` enforcement; HTTP error reporting; invalid vector
-  shape rejection; stable provider metadata revisions; and indexed result
-  ordering for providers that return embedding indexes.
-- Full gate passed after the adapter implementation: `npm run format:check`,
-  `npm run lint`, `npm run typecheck`, `npm test` with 92 tests,
-  `npm run build`, and `npm run eval` with 21 corpus questions, 21 sources,
-  synthetic retrieval 5/5, hybrid retrieval 2/2, table PDF 1/1, hard PDF 1/1,
-  and answer policy 1/1.
-
-Milestone 8 hybrid retrieval proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "RetrievalToolRuntime|athena server exposes"` passed 54 tests, including
-  vector-only semantic retrieval, FTS and vector score merging, chunk
-  deduplication, API trace score serialization, and UI trace rendering of
-  `searchScores`.
-- Built smoke passed: `node dist/cli/main.js index --root "$tmp" --json`
-  followed by compiled `RetrievalToolRuntime.searchDocs({ query: "login
-  credentials" })` returned `oauth.md` with `fts: 0`, positive vector score,
-  and merged score.
-
-Milestone 8 embedding progress proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "embedding batch progress|stores embedding vectors|records, skips"` passed
-  55 tests, including a 40-document fixture that reports `Embedding batch 1/2`
-  and `Embedding batch 2/2` on stderr while preserving final index output on
-  stdout.
-- Built smoke passed: `node dist/cli/main.js index --root "$tmp"` on a
-  40-document fixture reported two embedding batch lines on stderr and
-  `embedded: 40` on stdout.
-
-Milestone 8 neighboring context proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "RetrievalToolRuntime returns related neighboring chunks|RetrievalToolRuntime
-  searches"` passed 55 tests, including search-time neighboring chunk
-  expansion, `contextOfChunkId` trace output, and deduplication of expanded
-  results.
-- Built smoke passed: compiled `RetrievalToolRuntime.searchDocs({ query:
-  "p5w10", pathPrefix: "long.md", limit: 3 })` returned a neighboring result
-  with `contextOfChunkId` and a matching trace score marker.
-
-Milestone 8 ranking boost proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "applies identifier|RetrievalToolRuntime searches"` passed 56 tests,
-  including identifier, heading, source-title, and recently read source boost
-  fixtures.
-- Built smoke passed: compiled retrieval ranked `ranking/identifier-match.md`
-  first for `ATHENA_CACHE_DIR cache` and `ranking/recent-beta.md` first after
-  reading that source range.
-
-Milestone 8 generic penalty proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- Focused `npm test -- --test-name-pattern
-  "applies identifier|RetrievalToolRuntime searches"` passed 56 tests,
-  including a generic overview fixture that ranks below the specific
-  cache-directory document.
-- Built smoke passed: compiled retrieval ranked
-  `ranking/generic/specific-cache.md` above `ranking/generic/overview.md` for
-  `cache directory`.
-
-## Milestone 9: PDF RAG with `pdf-2-llm`
-
-Goal: use the published `pdf-2-llm` package to index PDFs with source maps,
-page regions, warnings, confidence, and citation-aware UI behavior.
-
-### 9.1 Dependency and Adapter
-
-- [x] Add `pdf-2-llm` as a package dependency.
-  Proof: lockfile and package manifest include the published npm package.
-- [x] Import the Node parser from `pdf-2-llm/node`.
-  Proof: parser adapter uses `convertPdfToMarkdown` from the Node export.
-- [x] Add parser version and option-hash tracking.
-  Proof: PDF source rows include parser package version and parser options hash.
-- [x] Add timeout and error handling for slow or malformed PDFs.
-  Proof: fixture with forced timeout records a parser failure without corrupting
-  the index.
-- [x] Preserve generated Markdown, source maps, document IR metadata, table
-  sidecars, extracted asset references, diagnostics, warning codes, and
-  confidence summaries when available.
-  Proof: parser artifact tests inspect stored records for each supported field.
-
-### 9.2 PDF Chunking and Storage
-
-- [x] Chunk generated PDF Markdown with page-aware locators.
-  Proof: chunks include source path, page number, Markdown range, and source-map
-  reference metadata.
-- [x] Store PDF parser artifacts in `parser_artifacts`.
-  Proof: database test retrieves generated Markdown and metadata by source ID.
-- [x] Store table sidecars and extracted asset references under `.athena/blobs/`.
-  Proof: fixture table or asset can be opened from stored artifact metadata.
-- [x] Store warning and confidence data on chunks.
-  Proof: low-confidence PDF fixture produces chunk warning/confidence fields.
-- [x] Rebuild PDF artifacts when parser version or options change.
-  Proof: changing PDF parser config invalidates only affected PDF sources.
-
-### 9.3 PDF Tooling and Citations
-
-- [x] Implement `readPdfSource`.
-  Proof: tool returns generated Markdown, page metadata, region metadata, and
-  warnings for a known PDF fixture.
-- [x] Extend `searchDocs` output for PDF chunks.
-  Proof: PDF search result includes page, heading, snippet, score, confidence,
-  warning codes, and locator.
-- [x] Validate PDF citations against known source paths.
-  Proof: unknown PDF path is rejected.
-- [x] Validate page numbers are inside document bounds.
-  Proof: out-of-range page citation is rejected.
-- [x] Validate page regions have source-map backing when required.
-  Proof: unsupported region citation is rejected or marked uncertain according
-  to config.
-- [x] Include confidence score and warning codes in PDF citation data.
-  Proof: answer citation rows include PDF confidence and warning metadata.
-
-### 9.4 PDF UI
-
-- [x] Add PDF citation rendering.
-  Proof: UI displays path, page number, confidence, and warning state.
-- [x] Add generated Markdown preview for PDF citations.
-  Proof: clicking a PDF citation opens the generated Markdown range.
-- [x] Add page-region metadata preview.
-  Proof: UI shows source-map coordinates or region details when available.
-- [x] Add uncertain citation display for low-confidence regions.
-  Proof: low-confidence fixture displays warning/confidence state near the
-  citation.
-- [x] Add table sidecar link behavior.
-  Proof: citation or preview can open table sidecar data when present.
-
-### 9.5 PDF Acceptance
-
-- [x] `pdf-2-llm` is installed from npm.
-  Proof: package manifest and lockfile reference the package.
-- [x] PDFs are converted through `pdf-2-llm/node`.
-  Proof: adapter test confirms the Node export is used.
-- [x] Generated Markdown and source maps are stored.
-  Proof: database/artifact test retrieves both for a fixture PDF.
-- [x] PDF citations include page and region metadata.
-  Proof: answer test verifies citation JSON shape.
-- [x] Low-confidence regions are visible in the UI.
-  Proof: browser test or screenshot fixture shows warning/confidence state.
-
-Milestone 9 dependency and adapter proof, 2026-07-03:
-
-- `npm view pdf-2-llm version name license main types exports bin dependencies --json`
-  confirmed `pdf-2-llm@1.1.0`, license `0BSD`, CLI binaries `pdf-2-llm`
-  and `pdf2md`, and a `./node` export with package types.
-- `npm install pdf-2-llm@1.1.0` added the published package to
-  `package.json` and `package-lock.json`; npm audit reported 0 vulnerabilities.
-- `npm run typecheck` passed.
-- Focused
-  `npm test -- --test-name-pattern "parsePdfFile|pdf-2-llm parser artifacts|classifySourceKind|discoverSources"`
-  passed 62 tests, including adapter use of a fake `convertPdfToMarkdown`
-  from the Node export shape, parser option hashing, timeout and malformed
-  converter errors, generated Markdown/source-map/document-IR/assets/
-  diagnostics/warnings/confidence artifact preservation, PDF source row parser
-  version and option hash storage, PDF chunk page/confidence locators, and
-  unchanged PDF skip behavior.
-- Built smoke passed: compiled `indexRoot({ root, pdfConverter })` from
-  `dist/indexer/index.js` wrote a `pdf-2-llm` source row, a page-4 PDF locator,
-  and `source-map` parser artifact for a temporary PDF fixture.
-
-Milestone 9 PDF chunk storage proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Focused
-  `npm test -- --test-name-pattern "parsePdfFile|pdf-2-llm parser artifacts"`
-  passed 62 tests, including PDF chunk locators with source path, page,
-  generated-Markdown start/end lines, source-map kind, source-map region
-  geometry, stable region ID, confidence, low-confidence warning codes, and a
-  parser-option change that reparses the unchanged PDF and updates the source
-  option hash.
-- Built smoke passed: compiled `indexRoot({ root, pdfConverter })` wrote a PDF
-  locator with page 4, generated-Markdown lines 1-3, `pdf-low-confidence`
-  warning code, source-map region geometry, and a stable `pdf_region_*` ID.
-
-Milestone 9 PDF blob sidecar proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Focused `npm test -- --test-name-pattern "pdf-2-llm parser artifacts"`
-  passed 62 tests, including a PDF table asset whose content is written under
-  `.athena/blobs/pdf-assets/<sourceId>/`, whose parser artifact stores
-  `athenaBlobPath` and `contentStored: true`, and whose inline asset content is
-  omitted from the database artifact payload.
-- Built smoke passed: compiled `indexRoot({ root, pdfConverter })` wrote
-  `tables/table-1.csv` under `.athena/blobs/pdf-assets/<sourceId>/` and the
-  `assets` artifact referenced that sidecar without inline `content`.
-
-Milestone 9 PDF read tooling proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- `npm run lint` passed.
-- `npm run format:check` passed.
-- `npm test` passed 63 tests.
-- `npm run build` passed.
-- Focused
-  `npm test -- --test-name-pattern "readPdfSource|generated PDF Markdown|toolSchemas|createModelToolDefinitions"`
-  passed 63 tests, including model tool schema exposure for `readPdfSource`,
-  PDF search results with snippet, score, page, confidence, warning codes, and
-  locator metadata, `readPdfSource` returning generated Markdown,
-  source-map entries, region metadata, warnings, confidence, assets, trace
-  locators, page-bound rejection, and source-map-backed region validation.
-- Built smoke passed: compiled `dist/indexer/index.js` and
-  `dist/retrieval/index.js` indexed a fake PDF converter result, then
-  `readPdfSource` returned page 2, one source-map entry, one asset, confidence
-  `0.5`, and a PDF trace locator.
-
-Milestone 9 PDF citation validation proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Focused
-  `npm test -- --test-name-pattern "citation|PDF citation|runAgentTurn persists PDF"`
-  passed 66 tests, including PDF citation parse/serialize, unknown PDF path
-  rejection, PDF answer citation persistence with page, region ID, confidence,
-  warning codes, and source-map region metadata, and existing text citation
-  validation.
-- `npm run lint` passed.
-- `npm run format:check` passed.
-- `npm test` passed 66 tests.
-- `npm run build` passed.
-- Built smoke passed: compiled `dist/index.js` exported
-  `parseAnswerCitations`, `parsePdfCitations`, `serializePdfCitation`, and
-  `renderTerminalCitations` for `[manual.pdf p.2#pdf_region_abc123]`.
-
-Milestone 9 PDF UI proof, 2026-07-03:
-
-- `npm run typecheck` passed.
-- Focused `npm test -- --test-name-pattern "PDF preview|local chat/search UI|server exposes"`
-  passed 67 tests, including the UI contract for PDF preview actions,
-  confidence/warning/region rendering hooks, sidecar asset controls, and the
-  server PDF preview/sidecar endpoints.
-- `npm run format:check` passed.
-- `npm run lint` passed.
-- `npm test` passed 67 tests.
-- `npm run build` passed.
-- Built smoke passed: compiled `dist/server/index.js` served the static UI,
-  `/api/pdf` returned page 2 with one asset and confidence metadata, and
-  `/api/pdf-asset` returned the table sidecar content.
-
-## Milestone 10: Evaluation Harness and Quality Gates
-
-Goal: prevent retrieval, citation, PDF, and cache correctness regressions.
-
-### 10.1 Evaluation Corpus
-
-- [x] Promote the Milestone 2 validation corpus into the evaluation harness.
-  Proof: eval configuration references corpus manifest IDs instead of ad hoc
-  fixture paths.
-- [x] Add synthetic Markdown corpus with known answers.
-  Proof: tests use the corpus for deterministic answer checks.
-- [x] Add realistic API-doc corpus.
-  Proof: retrieval tests include symbols, flags, methods, and concepts.
-- [x] Add ambiguous-heading corpus.
-  Proof: retrieval tests distinguish repeated heading names across files.
-- [x] Add table-heavy PDF fixture.
-  Proof: PDF tests cover table sidecar persistence and citation behavior.
-- [x] Add malformed, encrypted, or unsupported PDF fixture.
-  Proof: parser warning/failure behavior is tested without crashing.
-- [x] Add optional multilingual or RTL/CJK fixtures if target users require
-  them.
-  Proof: the current MVP corpus keeps one non-English French HTML fixture and
-  the corpus test requires non-English coverage; no RTL/CJK target-user
-  requirement is identified yet.
-
-### 10.2 Parser and Index Tests
-
-- [x] Test unchanged files are skipped.
-- [x] Test changed files are reindexed.
-- [x] Test deleted files are removed from the index.
-- [x] Test line ranges remain stable.
-- [x] Test PDF source maps point to valid page regions.
-- [x] Test warning and confidence fields survive storage.
-- [x] Test table sidecars can be opened from citations.
-- [x] Test migration upgrades preserve existing source and chunk rows.
-
-### 10.3 Retrieval Tests
-
-- [x] Test exact symbol query returns the symbol's defining section.
-- [x] Test semantic query returns the relevant conceptual section.
-- [x] Test hybrid retrieval outperforms FTS-only and vector-only on the
-  evaluation corpus.
-- [x] Test low-confidence PDF chunks are not over-promoted.
-- [x] Test source filters work by path and type.
-- [x] Test neighboring chunks are added without duplicating results.
-- [x] Test ranking remains deterministic for tied scores.
-
-### 10.4 Answer and Agent Tests
-
-- [x] Test answers include citations for doc-derived claims.
-- [x] Test unsupported claims are refused or marked unsupported.
-- [x] Test citations open the expected source range.
-- [x] Test follow-up questions use prior context without skipping fresh
-  retrieval.
-- [x] Test max-step guard stops runaway tool loops.
-- [x] Test invalid tool input cannot reach filesystem or database internals.
-- [x] Test final citation validation runs on every turn.
-
-### 10.5 Regression Commands
-
-- [x] Add `npm test`.
-  Proof: all unit and integration tests pass locally.
-- [x] Add `npm run typecheck`.
-  Proof: compiler passes without emitted output or with expected build output.
-- [x] Add `npm run lint`.
-  Proof: lint passes on source and tests.
-- [x] Add `npm run eval`.
-  Proof: evaluation harness reports parser, retrieval, answer, and PDF results.
-- [x] Add CI workflow if this repository will run hosted checks.
-  Proof: GitHub Actions or equivalent check runs the same local commands.
-
-Milestone 10 evaluation corpus harness proof, 2026-07-03:
-
-- `npm run typecheck` passed with `scripts/eval.ts` included through
-  `tsconfig.test.json`.
-- Focused `npm test -- --test-name-pattern "evaluation|corpus"` passed 70
-  tests, including eval-plan binding from `ground-truth.json` source IDs to
-  `manifest.json`, unknown source-ID diagnostics, and eval-plan rendering.
-- `npm run eval` passed offline and reported 20 questions, 19 healthy corpus
-  entries, 1 negative question, 8 document types, and redistribution counts of
-  `committed=6` and `download-only=13`.
-
-Milestone 10 synthetic retrieval eval proof, 2026-07-03:
-
-- `npm run typecheck` passed with the synthetic eval runner included.
-- Focused `npm test -- --test-name-pattern "evaluation|corpus"` passed 73
-  tests, including a temporary index built from supported synthetic Markdown and
-  text controls, exact-symbol retrieval with `admin-guide.md` as the top result,
-  semantic-control retrieval, and an explicit exclusion for the current
-  unsupported extensionless `cli-help` fixture.
-- `npm run eval` passed offline and reported `Synthetic retrieval: 4/4 passed`.
-
-Milestone 10 realistic API retrieval eval proof, 2026-07-03:
-
-- `npm run typecheck` passed with the real API eval runner included.
-- Focused `npm test -- --test-name-pattern "evaluation|corpus|inherited alias"`
-  passed 73 tests, including a cached Node fs API document indexed with Vite
-  and Prettier Markdown distractors, `fsPromises.access(path[, mode])` retrieval
-  with `node-fs-api.md` ranked first, and an embedding tokenizer regression for
-  inherited object keys such as `function`.
-- `npm run eval` passed offline and reported `API retrieval: 1/1 passed`.
-
-Milestone 10 ambiguous-heading eval proof, 2026-07-03:
-
-- Focused
-  `npm test -- --test-name-pattern "ambiguous heading|applies identifier|evaluation"`
-  passed 74 tests, including the corpus eval that ranks `admin-guide.md` first
-  over `user-guide.md` for the repeated `Tokens` heading and the retrieval
-  runtime fixture for the same ambiguity.
-- `npm run eval` passed offline and reported `Ambiguous headings: 1/1 passed`.
-
-Milestone 10 table-heavy PDF eval proof, 2026-07-03:
-
-- Fixed parser warning artifact IDs so repeated PDF warnings use an
-  ordinal-salted ID while preserving the stored `warning` artifact kind.
-- Focused
-  `npm test -- --test-name-pattern "table-heavy PDF|repeated PDF parser warnings|evaluation"`
-  passed 76 tests, including a real cached IRS W-4 index, PDF citation region
-  ID, source-map entries, `table.low_confidence`, confidence metadata,
-  persisted table sidecar blob path, nonzero sidecar bytes, and duplicate
-  warning payload storage.
-- `npm run typecheck` passed with the table-heavy PDF eval result types.
-- `npm run eval` passed offline and reported `Table PDF: 1/1 passed`.
-
-Milestone 10 hard PDF eval proof, 2026-07-03:
-
-- Focused `npm test -- --test-name-pattern "hard PDF|table-heavy PDF|evaluation"`
-  passed 77 tests, including the encrypted qpdf fixture indexing without a
-  failed index count, producing one document row, zero citeable chunks, parser
-  artifacts, `security.password_required`, `conversion.not_implemented`,
-  `pdf-low-confidence`, zero page count, and zero confidence.
-- `npm run typecheck` passed with the hard-PDF eval result types.
-- `npm run eval` passed offline and reported `Hard PDF: 1/1 passed`.
-
-Milestone 10 parser and index regression proof, 2026-07-03:
-
-- Focused
-  `npm test -- --test-name-pattern "migrateDatabase|records, skips, changes, deletes|line locators|PDF locators"`
-  passed 78 tests, including unchanged-file skips, changed-file reindexing,
-  deleted-source cleanup, deterministic chunk line locators, PDF page/region
-  locator metadata, and migration preservation of seeded source, document, and
-  chunk rows.
-- `npm run typecheck` passed with the migration preservation test.
-
-Milestone 10 retrieval regression proof, 2026-07-03:
-
-- Focused
-  `npm test -- --test-name-pattern "low-confidence PDF|applies identifier|generated PDF Markdown|searches, filters"`
-  passed 79 tests, including exact-symbol lookup for
-  `ATHENA_INDEX_BATCH_SIZE`, vector-only semantic lookup for login credentials,
-  path and kind filters, low-confidence PDF demotion behind an equivalent
-  Markdown source, neighbor deduplication, and deterministic ordering for tied
-  scores.
-- `npm run typecheck` passed with the low-confidence and deterministic ranking
-  tests.
-
-Milestone 10 hybrid retrieval comparison proof, 2026-07-03:
-
-- Added committed synthetic hybrid comparison fixtures under
-  `test/corpus/synthetic/hybrid/semantic/` and a ground-truth question that
-  requires vector recovery when FTS has no lexical hit.
-- Added evaluation-only `searchDocsWithMode` support for hybrid, FTS-only, and
-  vector-only retrieval without exposing those modes to model tool schemas.
-- Focused
-  `npm test -- --test-name-pattern "evaluation|hybrid retrieval|corpus manifest|renderEvaluationPlan"`
-  passed 82 tests, including the hybrid comparison reporting
-  `hybrid=2`, `fts=1`, and `vector=1`.
-- `npm run typecheck` passed with the hybrid comparison result types.
-- `npm run eval` reports `Hybrid retrieval: 2/2 passed (hybrid=2, fts=1, vector=1)`.
-
-Milestone 10 answer and agent proof, 2026-07-03:
-
-- Focused `npm test -- --test-name-pattern "runAgentTurn|parse.*Citations|terminal citations"`
-  passed 80 tests, including cited text and PDF answers, terminal citation
-  rendering, source-range validation, rejection of unseen/out-of-range
-  citations, rejection of substantive uncited answers, follow-up context with a
-  fresh search/read path, invalid tool input rejection before filesystem access,
-  and max-step tool-budget termination.
-- `npm run typecheck` passed with the follow-up retrieval regression.
-
-Milestone 10 regression command proof, 2026-07-03:
-
-- Added `.github/workflows/ci.yml` to run `npm ci`, `npm run format:check`,
-  `npm run lint`, `npm run typecheck`, `npm test`, `npm run build`, and
-  `npm run eval` on pushes to `main` and pull requests.
-- `npm run eval` passed offline and reported corpus health, synthetic
-  retrieval, API retrieval, ambiguous-heading retrieval, hybrid retrieval,
-  table-heavy PDF, encrypted hard PDF, and `Answer policy: 1/1 passed`.
-- The same local gate passed with `npm run format:check`, `npm run lint`,
-  `npm run typecheck`, `npm test`, `npm run build`, and `npm run eval`.
-
-## Milestone 11: MCP and External Agent Integration
-
-Goal: expose Athena's retrieval tools to external agents without letting them
-bypass Athena citation validation or read-only defaults.
-
-### 11.1 MCP Server
-
-- [x] Implement `athena mcp`.
-  Proof: command starts an MCP server over the chosen transport.
-- [x] Expose citation-aware retrieval tools only.
-  Proof: MCP tool list includes `searchDocs`, `readSourceRange`,
-  `readPdfSource`, `grepDocs`, `listDocs`, and `relatedChunks`.
-- [x] Keep MCP mode read-only by default.
-  Proof: tests verify no write/edit tools are exposed.
-- [x] Reuse the same index and tool validation as the UI/agent.
-  Proof: MCP tests call the same tool implementations.
-- [x] Add path and source validation for every MCP request.
-  Proof: path traversal and unknown source requests are rejected.
-
-### 11.2 External Agent Documentation
-
-- [x] Document OpenCode or coding-agent setup.
-  Proof: docs show how to start `athena mcp` and connect a client.
-- [x] Document that external agents consume Athena retrieval and do not own
-  Athena's agent loop.
-  Proof: integration docs explicitly state the boundary.
-- [x] Document trace compatibility between UI and MCP usage.
-  Proof: MCP calls can be inspected through stored run or retrieval trace data.
-- [x] Add manual smoke test for MCP search and source read.
-  Proof: committed test notes or automated script shows expected tool output.
-
-Milestone 11 MCP server proof, 2026-07-03:
-
-- Implemented `athena mcp` as a stdio JSON-RPC MCP server using protocol
-  version `2025-03-26`.
-- The server exposes only read-only retrieval tools:
-  `searchDocs`, `readSourceRange`, `readPdfSource`, `grepDocs`, `listDocs`,
-  and `relatedChunks`.
-- MCP tool calls dispatch to `RetrievalToolRuntime`, so they reuse the same
-  index, path checks, PDF read path, and retrieval behavior as the UI and agent.
-- Focused `npm test -- --test-name-pattern "MCP|mcp"` passed 85 tests,
-  including initialization, `tools/list`, read-only tool annotations, no
-  write/edit/delete tools, retrieval tool calls, unsafe path rejection, unknown
-  tool rejection, and the `runCli(["mcp", ...])` command path.
-- `npm run typecheck` passed with the MCP protocol and tool result types.
-
-Milestone 11 external agent documentation proof, 2026-07-03:
-
-- Added `docs/mcp-integration.md` with OpenCode local MCP configuration,
-  installed-package and development-checkout command examples, external-agent
-  boundary notes, MCP response trace shape, and a manual stdio smoke test for
-  `tools/list`, `searchDocs`, and `readSourceRange`.
-- Updated MCP tool results to include a `{ result, trace }` JSON envelope so
-  external-agent calls expose `seenChunkIds`, `readLocators`, and
-  `searchScores` consistently with Athena's runtime trace model.
-- Focused `npm test -- --test-name-pattern "MCP|mcp"` passed with trace
-  assertions for search and source-read calls.
-- Built CLI smoke passed: `node dist/cli/main.js mcp --root "$tmpdir"`
-  returned 4 JSON-RPC responses, 6 tools, `docs/admin-guide.md` as the top
-  search hit, and 1 read trace locator.
-
-Milestone 11 external agent provider option proof, 2026-07-05:
-
-- Added `agent.provider` with `opencode` as the default and `codex` as the
-  alternate provider.
-- Added `athena mcp --agent-provider <opencode|codex>` for one-off MCP setup.
-- Updated OpenCode and Codex MCP setup docs while preserving the same read-only
-  tool exposure for both providers.
-
-## Release Readiness Checklist
-
-Use this before publishing a public package or tagging a release.
-
-- [x] `athena --help` displays correct command name and subcommands.
-- [x] `npx @alextis59/athena` works from a fixture docs folder.
-- [x] Default startup builds or updates `.athena/index.sqlite`.
-- [x] Default startup skips unchanged files.
-- [x] Default startup opens or prints the local UI URL.
-- [x] `athena index` exits non-zero on parser, migration, or embedding failure.
-- [x] `athena serve` fails clearly when the index is missing unless `--build`
-  is provided.
-- [x] `athena ask` returns cited terminal answers.
-- [x] `athena doctor` reports database, schema, source count, embedding
-  provider, `pdf-2-llm`, warnings, model cache, and port state.
-- [x] Local embeddings are the default.
-- [x] Remote providers require explicit configuration.
-- [x] Markdown citations open exact line ranges.
-- [x] PDF citations include page, region, confidence, and warning metadata.
-- [x] Low-confidence PDF citations are visible as uncertain.
-- [x] Citation validator rejects unknown paths, out-of-range lines/pages,
-  unsupported PDF regions, and unseen sources.
-- [x] Conversation memory stores compact summaries, citations, timestamps, and
-  optional pinned sources.
-- [x] Validation corpus retrieval and health checks pass.
-- [x] Evaluation harness uses the validation corpus for every supported document
-  type.
-- [x] Full test, typecheck, lint, and eval commands pass.
-- [x] Package metadata, license, README, and documentation are ready for npm.
-
-Release readiness startup proof, 2026-07-03:
-
-- `athena --help` is covered by `runCli renders help successfully`, including
-  the default startup description and listed subcommands.
-- Bare `athena` is covered by `runCli default builds the index, skips unchanged
-  files, and prints the UI URL`, which creates `.athena/index.sqlite`, starts a
-  local server on an ephemeral port, and verifies a second run skips the
-  unchanged fixture file.
-- `athena serve` missing-index behavior is covered by `runCli serve requires an
-  existing index unless --build is provided`, which rejects a missing cache with
-  instructions to run `athena index` or `athena serve --build`, then verifies
-  `--build` indexes and prints the server URL.
-- Full gate passed after the startup behavior change: `npm run format:check`,
-  `npm run lint`, `npm run typecheck`, `npm test` with 94 tests,
-  `npm run build`, and `npm run eval` with 21 corpus questions, 21 sources,
-  synthetic retrieval 5/5, hybrid retrieval 2/2, table PDF 1/1, hard PDF 1/1,
-  and answer policy 1/1.
-
-Release readiness diagnostics proof, 2026-07-03:
-
-- `athena doctor` now reports source, document, chunk, and parser-warning
-  counts alongside database state, schema version, table availability,
-  `pdf-2-llm` availability, embedding provider/model, model cache state, and
-  server port availability.
-- Focused `npm test -- --test-name-pattern
-  "doctor|database and embedding failures|parser failures"` passed 95 tests,
-  including doctor output for missing, initialized, and indexed warning-bearing
-  fixtures; occupied-port JSON diagnostics; `athena index` non-zero exit on a
-  bad database path; no-network remote embedding failure via empty API key; and
-  failed-run recording for PDF parser failures.
-- Full gate passed after the diagnostics change: `npm run format:check`,
-  `npm run lint`, `npm run typecheck`, `npm test` with 95 tests,
-  `npm run build`, and `npm run eval` with 21 corpus questions, 21 sources,
-  synthetic retrieval 5/5, hybrid retrieval 2/2, table PDF 1/1, hard PDF 1/1,
-  and answer policy 1/1.
-
-Release readiness package proof, 2026-07-03:
-
-- README now documents installation, `npx @alextis59/athena`, CLI commands, common
-  options, local-first defaults, configuration, remote-provider opt-in,
-  citation guarantees, MCP integration, and development verification commands.
-- Added an ISC `LICENSE` file and package metadata for keywords and author.
-- Fixed the executable bin entrypoint so npm's `.bin/athena` symlink runs the
-  CLI instead of falling through silently.
-- Local package-run smoke from a fixture docs folder passed:
-  `npx --yes --package /home/alextis/Work/Git/alextis59/athena athena --help`
-  printed Athena help, and
-  `npx --yes --package /home/alextis/Work/Git/alextis59/athena athena index --json`
-  created `.athena/index.sqlite` with one discovered, parsed, embedded, and
-  committed source.
-- `npm pack --dry-run --json` reported `@alextis59/athena@1.0.0`, included `LICENSE`,
-  `README.md`, `docs`, `package.json`, and executable `dist/cli/main.js`, with
-  201 package entries and no bundled dependencies.
-- Remaining release checklist items are covered by the committed test and eval
-  gate: `athena ask` terminal citations, default local embeddings, remote
-  provider opt-in checks, Markdown and PDF citation validation, low-confidence
-  PDF uncertainty, conversation memory persistence, corpus health, and
-  evaluation coverage across AsciiDoc, extensionless, HTML, Markdown, MDX, PDF,
-  RST, and text documents.
-- Final package slice gate passed: `npm run format:check`, `npm run lint`,
-  `npm run typecheck`, `npm test` with 95 tests, `npm run build`,
-  `npm run eval`, and `npm pack --dry-run --json`.
-
-Scoped package publish proof, 2026-07-04:
-
-- `npm view athena name version description maintainers --json` confirmed the
-  unscoped `athena` package name is already owned on npm, so Athena publishes
-  as `@alextis59/athena` while preserving the installed binary name `athena`.
-- `npm view @alextis59/athena name version --json` returned npm `E404`, so the
-  scoped package target is not currently published on the public registry.
-- `npm pack --dry-run --json` reported `@alextis59/athena@1.0.0`, included
-  `LICENSE`, `README.md`, `docs`, `package.json`, and executable
-  `dist/cli/main.js`, with 201 package entries and no bundled dependencies.
-- Tarball command smoke passed from `alextis59-athena-1.0.0.tgz`:
-  `npx --yes --package ./alextis59-athena-1.0.0.tgz athena --help` printed
-  Athena help, and
-  `npx --yes --package ./alextis59-athena-1.0.0.tgz athena index --root <fixture> --json`
-  created `.athena/index.sqlite` with one discovered, parsed, embedded, and
-  committed source.
-- `npm publish --dry-run --json` passed with public access after running the
-  `prepublishOnly` gate, including format, lint, typecheck, 100 tests, offline
-  eval, and build.
-
-## Suggested Build Order
-
-1. Finish Milestone 0 so every later change has a repeatable command surface.
-2. Finish Milestone 1 before writing indexers, because parsers and retrieval
-   need stable persistence.
-3. Finish Milestone 2 before parser and retrieval work, because the validation
-   corpus should shape what "works" means.
-4. Finish Milestones 3 through 5 to prove local Markdown retrieval without any
-   model dependency.
-5. Finish Milestone 6 so citations and traces can be inspected visually.
-6. Finish Milestone 7 to complete the Markdown-only RAG MVP.
-7. Finish Milestone 8 to improve retrieval quality with local embeddings.
-8. Finish Milestone 9 to support PDFs through `pdf-2-llm`.
-9. Finish Milestone 10 before changing ranking, chunking, PDF parsing, or agent
-   citation behavior aggressively.
-10. Finish Milestone 11 after the internal tool and citation model is stable.
-
-## Per-PR Checklist
-
-For each implementation PR or commit sequence:
-
-- [x] Scope is tied to one milestone or a clearly named cross-cutting fix.
-- [x] Tests or manual proof cover the changed behavior.
-- [x] `npm run typecheck` passes when TypeScript code changes.
-- [x] `npm test` passes when behavior changes.
-- [x] `npm run eval` passes when retrieval, ranking, parsing, or citations
-  change.
-- [x] `docs/implementation-plan.md` is updated if checklist progress changes.
-- [x] User-facing behavior changes are reflected in README or docs.
-- [x] No generated `.athena/` cache, model cache, or local runtime artifact is
-  committed.