@alextis59/athena 1.0.0

package/docs/agentic-rag-study.md

@@ -0,0 +1,1056 @@
+# Agentic RAG Tool Design Study
+
+Date: 2026-07-03
+
+Source chat: https://chatgpt.com/share/6a47e6fe-1124-83ed-b2b6-8d3321dbcfcc
+
+This study turns the chat session into an implementation-oriented plan for a
+lightweight Node.js documentation RAG tool. It treats `athena` as the project
+that will own the CLI, index, retrieval engine, local chat UI, and optional
+agent integrations.
+
+Naming convention:
+
+- Project name: `athena`.
+- npm package name: `@alextis59/athena`.
+- CLI binary/command name: `athena`.
+- Package-run form: `npx @alextis59/athena`.
+- Local cache directory: `.athena/`.
+
+## Executive Summary
+
+The target product is a local-first documentation assistant that can be started
+from any documentation folder with a single Node command. It should discover
+docs, parse them, build a reusable cache, launch a local web chat UI, and answer
+questions with exact source references.
+
+The settled direction from the chat is:
+
+```txt
+Node.js + TypeScript CLI
+  + custom agent loop with no agent SDK dependency
+  + local web chat UI
+  + SQLite cache/index
+  + SQLite FTS5 lexical search
+  + embedded vector search
+  + local default embeddings
+  + optional cloud embeddings
+  + pdf-2-llm for PDF ingestion
+  + exact citations into source files and PDF page regions
+```
+
+The key architectural decision is that no SDK should handle Athena's agentic
+part. Athena should implement the agent loop directly: model-provider adapters,
+JSON-schema tools, tool execution, step limits, streaming, memory compaction, and
+citation validation. Existing agent hosts such as OpenCode can still be
+supported later through MCP or custom tools, but they should call Athena's
+retrieval API rather than replace it.
+
+The second major decision is that PDF handling must be treated as a serious
+ingestion subsystem. The chat originally explored building a custom PDF parser,
+then updated the plan once the user pointed to `pdf-2-llm`. The final design
+should use the published `pdf-2-llm` npm package as the canonical PDF parser and
+consume its Markdown, source maps, document IR, assets, warnings, diagnostics,
+confidence scores, and table sidecars.
+
+The third major decision is local-first embeddings. Athena should default to a
+small, fast, open-source embedding model executed locally, with cloud embeddings
+available as optional provider adapters. A good first default is
+`mixedbread-ai/mxbai-embed-xsmall-v1` via Transformers.js or ONNX. It is an
+open-source English retrieval model with 384-dimensional embeddings and 4096
+context support, which keeps a local SQLite vector index practical.
+
+## Product Goals
+
+Athena should optimize for documentation lookup in large local repositories,
+SDK docs, generated reference docs, PDFs, Markdown handbooks, and mixed-format
+technical archives.
+
+Primary goals:
+
+- Run from a folder with one command.
+- Build and reuse a local index.
+- Avoid reprocessing unchanged files.
+- Answer questions through a local chat UI.
+- Use retrieval tools rather than stuffing entire docs into the model context.
+- Return exact evidence references for every substantive answer.
+- Let users click citations to open the source file, line range, Markdown
+  anchor, or PDF page region.
+- Preserve useful follow-up context without replaying every intermediate tool
+  call.
+- Keep the default stack lightweight and Node-native.
+- Keep private documentation local by default.
+
+Non-goals for the first version:
+
+- Hosted multi-tenant service.
+- Distributed vector database.
+- Full replacement for a coding agent.
+- Fully autonomous repository editing.
+- Perfect PDF reconstruction for every malformed or scanned PDF.
+- Large framework dependency as the agent runtime.
+
+## Default Stack
+
+| Layer | Recommended Default | Notes |
+|---|---|---|
+| Runtime | Node.js + TypeScript | Matches the single-command requirement. |
+| CLI | `athena` | The project and command are named `athena`; `npx @alextis59/athena` is only the package-run form. |
+| Agent loop | Custom implementation | No SDK owns tool calling, step control, memory, or citation validation. |
+| Chat UI | Local web app served by the CLI | Gives control over citations and source previews. |
+| Cache/index | SQLite | One portable local database file. |
+| Lexical search | SQLite FTS5 | Good for exact API names, symbols, flags, and errors. |
+| Vector search | `sqlite-vec` or equivalent embedded SQLite vector extension | Keeps semantic search local and serverless. |
+| Default embeddings | `mixedbread-ai/mxbai-embed-xsmall-v1` | Small local open-source retrieval model. |
+| Embedding runtime | Transformers.js/ONNX adapter first | In-process local embedding path. |
+| Optional local embeddings | Ollama adapter | Useful for users already running local models. |
+| Optional cloud embeddings | OpenAI, Voyage, Cohere, Gemini adapters | Opt-in quality/cost tradeoff. |
+| PDF parser | `pdf-2-llm` | Use the published npm package and its structured result, not plain extracted text. |
+| Optional agent integration | MCP/OpenCode | Expose Athena's retrieval tools externally. |
+
+SQLite FTS5 is a native SQLite virtual table module for full-text search, so it
+is a good lexical-search base for a local index. `sqlite-vec` is a lightweight
+SQLite vector-search extension with Node bindings and no separate vector server.
+Those properties match the "single local command" requirement.
+
+## Alternatives and Tradeoffs from the Chat
+
+The chat evaluated several existing tools before settling on the custom core.
+Those tools are still useful context:
+
+- Vercel AI SDK was the initial recommendation for streaming, tool calling, and
+  multi-step agent behavior, but the final decision is to avoid SDK-managed
+  agentic behavior entirely. Athena should implement its own agent loop and use
+  third-party SDKs only as optional model-provider adapters when needed.
+- `remark`/`unified` remain good candidates for Markdown and MDX parsing because
+  their syntax-tree model can preserve positions for line-level citations.
+- LlamaIndex.TS and LangChain/LangGraph are useful references for agentic RAG
+  patterns, but they should not own Athena's runtime.
+- OpenAI File Search is a viable hosted alternative for teams that accept remote
+  storage and model-provider coupling, but it is not the default because Athena
+  is intended to be local-first and citation-controlled.
+- OpenCode is an integration target, not the product core. It can call Athena's
+  tools through MCP after the index and citation model are reliable.
+
+The final architecture keeps these options as reference points or adapters
+rather than foundations.
+
+## User Experience
+
+The intended workflow:
+
+```bash
+cd path/to/docs-or-project
+athena
+```
+
+For ad hoc use without installing the binary first:
+
+```bash
+npx @alextis59/athena
+```
+
+The command should:
+
+1. Discover supported files.
+2. Create or update `.athena/index.sqlite`.
+3. Hash files and skip unchanged content.
+4. Convert PDFs through `pdf-2-llm`.
+5. Parse Markdown, HTML, text, and code-adjacent docs.
+6. Chunk content with stable source locations.
+7. Build lexical and vector indexes.
+8. Launch a local web UI, for example `http://localhost:4317`.
+9. Let the user ask questions.
+10. Answer with citations that can be opened in context.
+
+Common CLI modes:
+
+```bash
+athena                 # index if needed, then launch UI
+athena index           # build/update index only
+athena serve           # serve existing index and UI
+athena ask "..."       # one-shot terminal question
+athena doctor          # inspect parser/index/runtime health
+athena mcp             # optional MCP server for external agents
+```
+
+## Repository and Cache Layout
+
+The local cache should live inside the indexed folder and be easy to remove:
+
+```txt
+.athena/
+  index.sqlite
+  blobs/
+    extracted-assets/
+    pdf-sidecars/
+  models/
+    embeddings/
+  logs/
+  runs/
+```
+
+The cache should include enough version metadata to invalidate only what is
+necessary:
+
+- Athena index schema version.
+- parser version by file type.
+- `pdf-2-llm` package version and option hash.
+- embedding provider, model, dimensions, pooling, normalization, and quantization.
+- chunking strategy version.
+- source file path, size, mtime, and content hash.
+- source-map hash for PDFs and generated Markdown.
+
+## Ingestion Pipeline
+
+The ingestion pipeline should be deterministic and resumable:
+
+```txt
+discover files
+  -> classify by type
+  -> hash source bytes
+  -> skip unchanged sources
+  -> parse to normalized document records
+  -> chunk with stable source anchors
+  -> embed chunks
+  -> write FTS rows
+  -> write vector rows
+  -> write source-map and asset metadata
+```
+
+Supported first-wave formats:
+
+- Markdown: `.md`, `.mdx`.
+- Plain text: `.txt`, `.rst`, `.adoc`.
+- HTML: `.html`, `.htm`.
+- PDF: `.pdf` through `pdf-2-llm`.
+- Code documentation files: `README`, `CHANGELOG`, `CONTRIBUTING`, package docs.
+
+File discovery should default to documentation-heavy paths and avoid expensive
+or generated directories:
+
+```txt
+include:
+  **/*.md
+  **/*.mdx
+  **/*.txt
+  **/*.rst
+  **/*.adoc
+  **/*.html
+  **/*.pdf
+
+exclude:
+  node_modules/**
+  .git/**
+  dist/**
+  build/**
+  coverage/**
+  .athena/**
+```
+
+The implementation should allow users to override both include and exclude
+rules in a config file.
+
+## PDF Ingestion with `pdf-2-llm`
+
+The chat's final PDF decision is clear: Athena should not implement its own PDF
+layout system as a parallel project. It should consume the existing
+`pdf-2-llm` output contract.
+
+`pdf-2-llm` is now a public npm package. As of July 4, 2026, the npm registry
+shows `pdf-2-llm@1.1.0` as `latest`, licensed `0BSD`, with CLI binaries named
+`pdf-2-llm` and `pdf2md`. The package exports Node, browser, worker, WASM, and
+schema entrypoints.
+
+Install it normally:
+
+```bash
+npm install pdf-2-llm
+```
+
+The package exposes:
+
+- Markdown output.
+- source-map entries back to PDF page regions.
+- document IR.
+- sidecar assets, including tables and extraction artifacts.
+- warning codes.
+- diagnostics.
+- confidence scores.
+- Node, browser, worker, CLI, schema, and WASM-related entrypoints.
+
+Athena should treat this as the default PDF dependency, not as a local workspace
+placeholder.
+
+Recommended conversion path:
+
+```ts
+import { readFile } from "node:fs/promises";
+import { convertPdfToMarkdown } from "pdf-2-llm/node";
+
+export async function parsePdf(path: string) {
+  const bytes = await readFile(path);
+  return convertPdfToMarkdown(bytes, {
+    markdown: { pageAnchors: true },
+    tables: { csvSidecars: true },
+    security: { timeoutMs: 30_000 }
+  });
+}
+```
+
+Athena should store the full structured result, not just the Markdown string.
+The source map and IR are essential to exact citations, PDF highlights, table
+references, extraction-confidence ranking, and future debugging.
+
+## PDF Alternatives and Fallback Strategy
+
+Before the `pdf-2-llm` update, the chat explored a layered PDF strategy:
+
+- `pdfjs` as a JavaScript-friendly baseline engine;
+- MuPDF as a stronger native extraction engine when packaging complexity is
+  acceptable;
+- a robust mode with OCR and heavier layout recovery for difficult PDFs;
+- Docling and PyMuPDF4LLM as references or optional plugins, not default
+  dependencies.
+
+After the user identified `pdf-2-llm`, the right move is to use that library as
+the primary parser. The fallback concepts still matter operationally:
+
+- low-confidence `pdf-2-llm` output should trigger warnings in the UI;
+- scanned PDFs should route to OCR-capable workflows when available;
+- parser diagnostics should be stored so users can understand extraction limits;
+- Athena should keep the parser adapter boundary clean enough to add a MuPDF,
+  Docling, or PyMuPDF4LLM plugin later if `pdf-2-llm` cannot handle a corpus.
+
+## PDF Citation Model
+
+For Markdown and text files, a citation can be:
+
+```json
+{
+  "kind": "text",
+  "path": "docs/example.md",
+  "startLine": 42,
+  "endLine": 55,
+  "snippetHash": "..."
+}
+```
+
+For PDFs, a citation should use both the generated Markdown location and the PDF
+source-map location:
+
+```json
+{
+  "kind": "pdf",
+  "path": "manual.pdf",
+  "markdownOffsetStart": 1204,
+  "markdownOffsetEnd": 1422,
+  "page": 17,
+  "regions": [
+    { "x": 72, "y": 144, "width": 440, "height": 88 }
+  ],
+  "confidence": 0.91,
+  "warningCodes": []
+}
+```
+
+The UI should show both:
+
+- an answer-side citation label, such as `[manual.pdf p.17]`;
+- a source panel with the exact converted Markdown snippet and a page-region
+  preview when possible.
+
+The citation validator should reject answer claims that cannot be tied to a
+retrieved chunk or source range.
+
+## Chunking Strategy
+
+Chunking should be source-aware rather than purely token-count based.
+
+Markdown and HTML chunking:
+
+- Prefer heading-bounded chunks.
+- Keep stable heading paths.
+- Preserve line ranges.
+- Keep code blocks intact.
+- Avoid splitting tables unless they are too large.
+- Store neighboring chunk IDs for context expansion.
+
+PDF chunking:
+
+- Chunk from `pdf-2-llm` Markdown and IR.
+- Preserve PDF page anchors.
+- Keep tables, captions, figures, and paragraphs coherent.
+- Use source-map entries to map chunk spans back to page regions.
+- Mark chunks derived from low-confidence extraction.
+- Keep sidecar references for CSV tables and asset outputs.
+
+Chunk size guidelines:
+
+- Target 300 to 900 tokens for normal prose.
+- Allow larger chunks for API reference sections with dense examples.
+- Keep a small overlap only when it preserves sentence or list continuity.
+- Store both raw text and normalized searchable text.
+
+## Index Schema
+
+A practical first schema can stay small:
+
+```sql
+create table sources (
+  id integer primary key,
+  path text not null unique,
+  kind text not null,
+  sha256 text not null,
+  size_bytes integer not null,
+  mtime_ms integer not null,
+  parser_name text not null,
+  parser_version text not null,
+  parser_options_hash text not null,
+  indexed_at text not null
+);
+
+create table documents (
+  id integer primary key,
+  source_id integer not null references sources(id),
+  title text,
+  metadata_json text not null
+);
+
+create table chunks (
+  id integer primary key,
+  document_id integer not null references documents(id),
+  source_id integer not null references sources(id),
+  ordinal integer not null,
+  heading_path text,
+  text text not null,
+  normalized_text text not null,
+  token_count integer not null,
+  locator_json text not null,
+  extraction_confidence real,
+  warning_codes_json text not null
+);
+
+create virtual table chunks_fts using fts5(
+  normalized_text,
+  heading_path,
+  content='chunks',
+  content_rowid='id'
+);
+
+create table embeddings (
+  chunk_id integer primary key references chunks(id),
+  provider text not null,
+  model text not null,
+  dimensions integer not null,
+  config_hash text not null,
+  vector_blob blob not null
+);
+```
+
+If `sqlite-vec` is used, vector storage can be implemented with a `vec0` virtual
+table or a compatible extension-specific table. Keep the relational metadata in
+normal SQLite tables so vector-extension migrations do not force a full metadata
+rewrite.
+
+## Embedding Model Recommendation
+
+The default should be local, open-source, and fast. The chat converged on
+`mixedbread-ai/mxbai-embed-xsmall-v1` as a strong default because it is small,
+retrieval-oriented, 384-dimensional, and supports 4096 context. Those properties
+are attractive for a local SQLite-backed index:
+
+- 384 dimensions reduces index size and speeds similarity scoring.
+- 4096 context gives chunking more headroom for long documentation sections.
+- Local inference avoids sending private docs to a cloud service.
+- Apache 2.0 licensing is easier for broad reuse than unclear model terms.
+- Matryoshka and binary-quantization support leave room for future size/speed
+  experiments.
+
+Recommended default embedding config:
+
+```yaml
+embeddings:
+  provider: transformers-js
+  model: mixedbread-ai/mxbai-embed-xsmall-v1
+  dimensions: 384
+  pooling: mean
+  normalize: true
+  batchSize: 32
+```
+
+Provider alternatives:
+
+| Provider | Role | Notes |
+|---|---|---|
+| `transformers-js` | Default in-process local runtime | Best default for single-command Node usage. |
+| `ollama` | Optional local service runtime | Useful when users already manage local models with Ollama. |
+| `openai` | Optional cloud adapter | Simple external quality baseline. |
+| `voyage` | Optional cloud adapter | Consider for code-heavy docs and retrieval quality experiments. |
+| `cohere` | Optional cloud adapter | Consider for multilingual or enterprise setups. |
+| `gemini` | Optional cloud adapter | Consider when already using Google infrastructure. |
+
+Ollama should be treated as an optional local provider, not the default, because
+it requires a separate local service. It is still valuable: Ollama documents an
+embedding API and currently recommends models such as `embeddinggemma`,
+`qwen3-embedding`, and `all-minilm`.
+
+Cloud embeddings should always be opt-in. The CLI should require explicit
+configuration before sending document text to a remote service.
+
+## Embedding Survey Notes
+
+The chat's final recommendation favored `mxbai-embed-xsmall-v1`, but the model
+survey also identified alternatives worth keeping in the design:
+
+- `all-MiniLM` is a small, established local baseline and is especially useful
+  through Ollama or ONNX distributions.
+- BGE-M3 is a stronger multilingual option with dense, sparse, and multi-vector
+  retrieval behavior, but it is heavier than the default.
+- Qwen3 embedding models are promising quality-oriented local options,
+  especially for multilingual and code-heavy retrieval, but should be optional.
+- EmbeddingGemma is a useful local Ollama option when users already run Ollama.
+- `nomic-embed-text` is another local-friendly option with long-context
+  retrieval variants.
+- FastEmbed-style runtimes are relevant for speed, but Athena should keep a
+  provider interface so runtime choice can change without rewriting indexing.
+- Matryoshka representation support and binary quantization are future
+  optimization paths for smaller indexes and faster local retrieval.
+
+The rule should be empirical: ship one fast local default, then compare optional
+providers on Athena's own documentation-evaluation harness.
+
+## Embedding Provider Interface
+
+Athena should isolate embeddings behind a small interface:
+
+```ts
+export type EmbeddingInput = {
+  id: string;
+  text: string;
+  metadata?: Record<string, unknown>;
+};
+
+export type EmbeddingVector = {
+  id: string;
+  vector: Float32Array;
+  dimensions: number;
+};
+
+export interface EmbeddingProvider {
+  readonly name: string;
+  readonly model: string;
+  readonly dimensions: number;
+  readonly configHash: string;
+
+  embedBatch(inputs: EmbeddingInput[]): Promise<EmbeddingVector[]>;
+}
+```
+
+Embedding cache invalidation should include:
+
+- provider name;
+- model name;
+- model revision or digest when available;
+- dimensions;
+- pooling;
+- normalization;
+- quantization settings;
+- chunk text hash;
+- chunking strategy version.
+
+Changing any of these should invalidate only affected embeddings, not the whole
+source parse.
+
+## Retrieval Design
+
+The retrieval engine should use hybrid search:
+
+1. Rewrite or decompose the user's question when useful.
+2. Run FTS5 search for exact terms.
+3. Run vector search for semantic matches.
+4. Run targeted grep for identifiers, package names, CLI flags, and errors.
+5. Merge and deduplicate results by source and chunk.
+6. Boost heading-path matches and exact symbol matches.
+7. Optionally apply reranking to the merged candidate set when the corpus or
+   model provider justifies the extra latency.
+8. Penalize low-confidence PDF chunks unless the query explicitly targets them.
+9. Expand around top chunks to include neighboring context.
+10. Validate that final answer citations point to retrieved evidence.
+
+This is the core reason Athena should not depend only on an external coding
+agent's native grep/read tools. Documentation RAG needs indexing metadata,
+source-map awareness, chunk scoring, source confidence, and citation validation.
+
+## Agent Tool Design
+
+The agent should see a small set of precise tools.
+
+### `searchDocs`
+
+Purpose: hybrid search over indexed docs.
+
+Input:
+
+```json
+{
+  "query": "string",
+  "filters": {
+    "pathPrefix": "string",
+    "kind": "markdown | pdf | html | text"
+  },
+  "limit": 10
+}
+```
+
+Output:
+
+```json
+{
+  "results": [
+    {
+      "chunkId": 123,
+      "path": "docs/api.md",
+      "headingPath": "Authentication > Tokens",
+      "snippet": "...",
+      "score": 0.82,
+      "locator": { "startLine": 10, "endLine": 22 }
+    }
+  ]
+}
+```
+
+### `readSourceRange`
+
+Purpose: read an exact source range from a non-PDF document.
+
+Input:
+
+```json
+{
+  "path": "docs/api.md",
+  "startLine": 10,
+  "endLine": 40
+}
+```
+
+### `readPdfSource`
+
+Purpose: read converted PDF Markdown plus original page-region metadata.
+
+Input:
+
+```json
+{
+  "path": "manual.pdf",
+  "page": 17,
+  "regionId": "..."
+}
+```
+
+### `grepDocs`
+
+Purpose: exact string or regex search for identifiers.
+
+Input:
+
+```json
+{
+  "pattern": "AUTH_TOKEN",
+  "caseSensitive": false,
+  "limit": 20
+}
+```
+
+### `listDocs`
+
+Purpose: inspect indexed sources and narrow search scope.
+
+Input:
+
+```json
+{
+  "pathPrefix": "docs/",
+  "kind": "pdf"
+}
+```
+
+### `relatedChunks`
+
+Purpose: fetch neighboring or related chunks after a promising hit.
+
+Input:
+
+```json
+{
+  "chunkId": 123,
+  "before": 1,
+  "after": 2
+}
+```
+
+## Custom Agent Loop
+
+The agent loop must be small, explicit, and Athena-owned. This is not an SDK
+integration point. It is the core control surface for tool selection, step
+limits, memory policy, streaming, and citation enforcement:
+
+```ts
+export type ModelMessage =
+  | { role: "system"; content: string }
+  | { role: "user"; content: string }
+  | { role: "assistant"; content: string; toolCalls?: ToolCall[] }
+  | { role: "tool"; toolCallId: string; content: unknown };
+
+export type ToolDef = {
+  name: string;
+  description: string;
+  inputSchema: unknown;
+  execute(input: unknown): Promise<unknown>;
+};
+
+export interface ModelProvider {
+  streamResponse(input: {
+    model: string;
+    messages: ModelMessage[];
+    tools: ToolDef[];
+    temperature?: number;
+  }): AsyncIterable<ModelEvent>;
+}
+```
+
+Required loop behavior:
+
+- enforce a max-step count;
+- stream assistant output to the UI;
+- execute only registered tools;
+- validate tool inputs before execution;
+- record tool traces for debugging;
+- perform conversation compaction between turns;
+- store final answer citations;
+- reject or warn on unsupported citations;
+- expose deterministic failure messages when retrieval is insufficient.
+
+The system prompt should instruct the agent to:
+
+- search before answering factual doc questions;
+- cite every claim that depends on documentation;
+- prefer exact source ranges over broad summaries;
+- say when the index has no evidence;
+- use follow-up memory only as conversational context, not as documentary proof.
+
+## Follow-Up Memory
+
+The chat identified a useful memory policy: keep questions, final answers, and
+citations, but avoid replaying every intermediate search step into the model
+context.
+
+A stored turn can look like:
+
+```json
+{
+  "question": "How do I configure OAuth?",
+  "answerSummary": "Use providers.github and set callbackUrl...",
+  "citations": [
+    { "path": "docs/auth.md", "startLine": 40, "endLine": 58 }
+  ],
+  "createdAt": "2026-07-03T12:00:00Z"
+}
+```
+
+On follow-up, the model can receive:
+
+- recent user questions;
+- short answer summaries;
+- citation identifiers;
+- any user-pinned sources;
+- fresh retrieval results for the new question.
+
+It should not receive full prior tool traces unless the user asks for audit
+details.
+
+## Local Web UI
+
+The UI should be simple, but citation-first:
+
+- chat column;
+- retrieval trace drawer;
+- source preview panel;
+- clickable citations;
+- line-highlighted Markdown/text viewer;
+- PDF page preview with highlighted regions when available;
+- index status indicator;
+- parser warning badges;
+- "open in editor" links for local files when feasible;
+- controls for reindexing and source filtering.
+
+Answer citations should be visible inline and collected at the bottom of each
+answer. Clicking a citation should open the exact source range, not only the
+file.
+
+## Optional MCP and OpenCode Integration
+
+OpenCode or another agent should be an optional consumer of Athena's index.
+
+The best integration is an MCP server exposing:
+
+- `searchDocs`;
+- `readSourceRange`;
+- `readPdfSource`;
+- `grepDocs`;
+- `listDocs`;
+- `relatedChunks`.
+
+This lets external agents use Athena as a specialized documentation search
+backend while Athena retains responsibility for indexing, ranking, parsing, and
+citations.
+
+Athena should also support a normal local chat UI because many documentation
+queries do not require a coding agent.
+
+## Configuration
+
+Example `athena.config.ts`:
+
+```ts
+import { defineConfig } from "@alextis59/athena";
+
+export default defineConfig({
+  index: {
+    include: ["**/*.md", "**/*.mdx", "**/*.pdf", "**/*.html"],
+    exclude: ["node_modules/**", "dist/**", ".git/**"]
+  },
+  embeddings: {
+    provider: "transformers-js",
+    model: "mixedbread-ai/mxbai-embed-xsmall-v1",
+    dimensions: 384,
+    pooling: "mean",
+    normalize: true
+  },
+  pdf: {
+    parser: "pdf-2-llm",
+    pageAnchors: true,
+    csvSidecars: true,
+    timeoutMs: 30_000,
+    minConfidenceForAutoCitation: 0.65
+  },
+  server: {
+    host: "127.0.0.1",
+    port: 4317
+  }
+});
+```
+
+Cloud providers should require explicit configuration:
+
+```ts
+embeddings: {
+  provider: "openai",
+  model: "text-embedding-3-small",
+  allowRemoteDocumentText: true
+}
+```
+
+The explicit `allowRemoteDocumentText` flag prevents accidental leakage of
+private docs.
+
+## Project Structure
+
+A practical implementation layout:
+
+```txt
+athena/
+  src/
+    cli/
+      main.ts
+      commands/
+    server/
+      app.ts
+      routes/
+    ui/
+      ...
+    indexer/
+      discover.ts
+      pipeline.ts
+      chunking.ts
+      cache.ts
+    parsers/
+      markdown.ts
+      html.ts
+      text.ts
+      pdf2llm.ts
+    embeddings/
+      provider.ts
+      transformers-js.ts
+      ollama.ts
+      openai.ts
+    retrieval/
+      fts.ts
+      vector.ts
+      hybrid.ts
+      rerank.ts
+    agent/
+      provider.ts
+      loop.ts
+      tools.ts
+      citations.ts
+      memory.ts
+    storage/
+      db.ts
+      migrations/
+    mcp/
+      server.ts
+  docs/
+    agentic-rag-study.md
+```
+
+## MVP Plan
+
+### Phase 1: Markdown-Only Local RAG
+
+Deliver:
+
+- CLI startup.
+- file discovery and hashing.
+- SQLite metadata tables.
+- Markdown parsing with line ranges.
+- FTS5 lexical search.
+- local web chat UI.
+- basic custom agent loop.
+- `searchDocs` and `readSourceRange`.
+- citation validation for Markdown line ranges.
+
+This phase proves the end-to-end user experience with the least parser risk.
+
+### Phase 2: Local Embeddings and Hybrid Search
+
+Deliver:
+
+- Transformers.js embedding provider.
+- `mxbai-embed-xsmall-v1` default.
+- embedding cache invalidation.
+- vector storage.
+- hybrid FTS + vector retrieval.
+- neighbor chunk expansion.
+- answer trace display in the UI.
+
+This phase proves answer quality beyond exact keyword matching.
+
+### Phase 3: PDF Ingestion with `pdf-2-llm`
+
+Deliver:
+
+- `pdf-2-llm` parser adapter.
+- structured result persistence.
+- PDF Markdown chunking.
+- PDF source-map citations.
+- PDF warning/confidence display.
+- `readPdfSource` tool.
+- table sidecar handling.
+
+This phase makes Athena useful for real documentation archives.
+
+### Phase 4: Quality Gates and Evaluation
+
+Deliver:
+
+- fixed evaluation corpus.
+- retrieval recall tests.
+- citation correctness tests.
+- PDF source-map integrity tests.
+- stale-cache invalidation tests.
+- answer faithfulness checks.
+- regression CLI for local runs.
+
+This phase prevents the RAG system from becoming a demo that cannot be trusted.
+It should include an evaluation harness that can be run before changing
+chunking, embeddings, ranking, PDF parsing, or citation validation.
+
+### Phase 5: MCP/OpenCode Bridge
+
+Deliver:
+
+- MCP server wrapping Athena tools.
+- external-agent setup docs.
+- trace compatibility between UI and MCP usage.
+- read-only default permissions.
+
+This phase lets coding agents consume the index without taking over the core.
+
+## Evaluation Plan
+
+Athena needs tests at three levels.
+
+Parser/index tests:
+
+- unchanged files are skipped;
+- changed files are reindexed;
+- line ranges remain stable;
+- PDF source maps point to valid page regions;
+- warning and confidence fields survive storage;
+- table sidecars can be opened from citations.
+
+Retrieval tests:
+
+- exact symbol query returns the symbol's defining section;
+- semantic query returns relevant conceptual section;
+- hybrid retrieval outperforms FTS-only and vector-only on a small corpus;
+- low-confidence PDF chunks are not over-promoted;
+- source filters work by path and type.
+
+Answer tests:
+
+- answer includes citations for doc-derived claims;
+- unsupported claims are rejected or marked as unsupported;
+- citations open the expected source range;
+- follow-up questions use prior context without skipping fresh retrieval;
+- max-step guard stops runaway tool loops.
+
+Evaluation datasets should include:
+
+- synthetic Markdown docs with known answers;
+- realistic API docs;
+- docs with repeated terms and ambiguous headings;
+- table-heavy PDFs;
+- scanned or OCR-planned PDFs;
+- malformed or encrypted PDFs;
+- multilingual or RTL/CJK PDFs if those are target users.
+
+## Risks and Mitigations
+
+| Risk | Mitigation |
+|---|---|
+| PDF source maps are noisy | Display confidence, warnings, and snippets; gate citations below a threshold. |
+| Local embeddings are lower quality than cloud | Keep cloud adapters opt-in and add corpus-level evals. |
+| Vector extension portability issues | Keep vector metadata separate and support rebuilds. |
+| Agent invents unsupported claims | Citation validator and retrieval-required prompt policy. |
+| Cache invalidation bugs | Include parser, chunker, model, and source hashes in cache keys. |
+| Large docs make indexing slow | Incremental hashing, batching, progress UI, and resumable indexing. |
+| Context flooding in follow-ups | Store summaries and citations, not all tool traces. |
+| External-agent integration bypasses citations | Expose only citation-aware retrieval tools through MCP. |
+
+## Final Recommendation
+
+Build Athena as a local Node/TypeScript documentation RAG system with a custom,
+SDK-free agent loop and a specialized retrieval engine. Use SQLite for the
+cache, FTS5 for exact lexical search, an embedded vector extension for semantic
+search, Transformers.js/ONNX with `mxbai-embed-xsmall-v1` for default local
+embeddings, and the published `pdf-2-llm` npm package as the PDF ingestion core.
+
+Do not use OpenCode, LangChain, LlamaIndex, Vercel AI SDK, or any other agent
+SDK to handle Athena's agentic behavior. They can be useful references, optional
+integrations, or model-provider adapters, but the hard part of this product is
+not "calling a model." The hard parts are stable indexing, source-aware
+chunking, exact citation mapping, PDF extraction quality, cache correctness, and
+answer validation. Those should remain Athena-owned.
+
+## References
+
+- Source chat: https://chatgpt.com/share/6a47e6fe-1124-83ed-b2b6-8d3321dbcfcc
+- `pdf-2-llm` npm package: https://www.npmjs.com/package/pdf-2-llm
+- `pdf-2-llm`: https://github.com/alextis59/pdf-2-llm
+- `mixedbread-ai/mxbai-embed-xsmall-v1`: https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1
+- Transformers.js: https://huggingface.co/docs/transformers.js/en/index
+- Ollama embeddings: https://docs.ollama.com/capabilities/embeddings
+- SQLite FTS5: https://www.sqlite.org/fts5.html
+- `sqlite-vec`: https://alexgarcia.xyz/sqlite-vec/