@alextis59/athena 1.2.0 → 1.3.0

package/docs/agentic-rag-specifications.md

@@ -1,670 +0,0 @@
-# Agentic RAG Tool Specifications
-
-Date: 2026-07-03
-
-Related study: [Agentic RAG Tool Design Study](./agentic-rag-study.md)
-
-## Purpose
-
-Athena is a local-first documentation RAG tool. It indexes documentation in the
-current folder, launches a local chat UI, and answers questions with exact
-citations into the original sources.
-
-This specification converts the study into concrete product and engineering
-requirements.
-
-## Naming
-
-- Project name: `athena`.
-- npm package name: `@alextis59/athena`.
-- CLI binary/command name: `athena`.
-- Package-run form: `npx @alextis59/athena`.
-- Local cache directory: `.athena/`.
-
-All product documentation and command examples should treat `athena` as the
-canonical project and command name. `npx @alextis59/athena` is only the
-no-install npm invocation form.
-
-## Final Architecture Decisions
-
-1. Athena owns the agentic loop.
-   No agent SDK handles tool selection, tool execution, step control, memory
-   policy, streaming, or citation validation.
-
-2. `pdf-2-llm` is the canonical PDF parser.
-   PDF ingestion uses the published npm package `pdf-2-llm`, not a local
-   workspace-only dependency.
-
-3. Local-first is the default.
-   Indexes, embeddings, and chat UI should run locally unless the user
-   explicitly opts into remote model or embedding providers.
-
-4. Citations are first-class data.
-   Every substantive answer must be grounded in source ranges or PDF page
-   regions that the UI can open.
-
-5. External agents are integrations.
-   OpenCode, MCP clients, LangChain, LlamaIndex, Vercel AI SDK, and similar
-   tools may integrate with Athena, but none of them should own Athena's core
-   agent behavior.
-
-## Functional Requirements
-
-### FR-1: Single Command Startup
-
-Athena must support a command that can be run inside any documentation folder:
-
-```bash
-athena
-```
-
-For ad hoc execution without installing the binary first, users may run
-`npx @alextis59/athena`.
-
-The command must:
-
-- discover supported documentation files;
-- create or update a local `.athena/index.sqlite` cache;
-- skip unchanged files;
-- start a local web UI;
-- expose a chat interface connected to the local index.
-
-### FR-2: Index-Only Mode
-
-Athena must support indexing without starting the UI:
-
-```bash
-athena index
-```
-
-This command must exit non-zero when parsing, database migration, or embedding
-generation fails.
-
-### FR-3: Serve Existing Index
-
-Athena must support serving a previously built index:
-
-```bash
-athena serve
-```
-
-If the index is missing, the command should either fail with a clear message or
-offer an explicit `--build` mode.
-
-### FR-4: One-Shot Ask Mode
-
-Athena should support terminal questions:
-
-```bash
-athena ask "How do I configure OAuth?"
-```
-
-The answer must include citations in terminal-readable form.
-
-### FR-5: Doctor Mode
-
-Athena should expose a diagnostic command:
-
-```bash
-athena doctor
-```
-
-It should report:
-
-- database status;
-- schema version;
-- indexed source count;
-- embedding provider status;
-- `pdf-2-llm` package availability;
-- parser warnings summary;
-- model cache status;
-- UI port availability.
-
-### FR-6: MCP Server Mode
-
-Athena should expose an optional MCP server:
-
-```bash
-athena mcp
-```
-
-The MCP server must expose citation-aware retrieval tools only. It must not
-allow external agents to bypass Athena's source validation.
-
-## Supported Inputs
-
-### Initial File Types
-
-Athena must support:
-
-- Markdown: `.md`, `.mdx`;
-- text: `.txt`;
-- reStructuredText: `.rst`;
-- AsciiDoc: `.adoc`;
-- HTML: `.html`, `.htm`;
-- PDF: `.pdf`;
-- extensionless documentation files such as `README`, `CHANGELOG`,
-  `CONTRIBUTING`, and `LICENSE`.
-
-### Default Exclusions
-
-Athena must exclude:
-
-- `.git/**`;
-- `node_modules/**`;
-- `.athena/**`;
-- `dist/**`;
-- `build/**`;
-- `coverage/**`;
-- binary files not explicitly supported.
-
-## Configuration
-
-Athena should load optional config from:
-
-- `athena.config.ts`;
-- `athena.config.mjs`;
-- `athena.config.json`.
-
-Example:
-
-```ts
-import { defineConfig } from "@alextis59/athena";
-
-export default defineConfig({
-  index: {
-    include: ["**/*.md", "**/*.mdx", "**/*.pdf", "**/*.html"],
-    exclude: ["node_modules/**", "dist/**", ".git/**"]
-  },
-  embeddings: {
-    provider: "transformers-js",
-    model: "mixedbread-ai/mxbai-embed-xsmall-v1",
-    dimensions: 384,
-    pooling: "mean",
-    normalize: true
-  },
-  agent: {
-    provider: "opencode"
-  },
-  pdf: {
-    parser: "pdf-2-llm",
-    pageAnchors: true,
-    csvSidecars: true,
-    timeoutMs: 30_000,
-    minConfidenceForAutoCitation: 0.65
-  },
-  server: {
-    host: "127.0.0.1",
-    port: 4317
-  }
-});
-```
-
-## PDF Ingestion Specification
-
-### Package
-
-Athena must use the public npm package:
-
-```bash
-npm install pdf-2-llm
-```
-
-The Node parser import should use:
-
-```ts
-import { convertPdfToMarkdown } from "pdf-2-llm/node";
-```
-
-### PDF Parser Output
-
-Athena must preserve these `pdf-2-llm` outputs when available:
-
-- Markdown;
-- source-map entries;
-- page regions;
-- document IR;
-- table sidecars;
-- extracted assets;
-- warning codes;
-- diagnostics;
-- confidence scores.
-
-### PDF Storage Requirements
-
-For every parsed PDF, Athena must store:
-
-- original file path;
-- source hash;
-- parser package version;
-- parser options hash;
-- generated Markdown;
-- source-map metadata;
-- document IR metadata;
-- warning summary;
-- confidence summary;
-- sidecar asset references.
-
-### PDF Citation Requirements
-
-PDF citations must include:
-
-- source file path;
-- page number;
-- page region coordinates when available;
-- generated Markdown offset or line range;
-- confidence score when available;
-- warning codes that affect the cited region.
-
-Example:
-
-```json
-{
-  "kind": "pdf",
-  "path": "manual.pdf",
-  "page": 17,
-  "regions": [
-    { "x": 72, "y": 144, "width": 440, "height": 88 }
-  ],
-  "markdownOffsetStart": 1204,
-  "markdownOffsetEnd": 1422,
-  "confidence": 0.91,
-  "warningCodes": []
-}
-```
-
-### PDF Warning Behavior
-
-Athena must not silently hide extraction uncertainty.
-
-If a cited PDF region has warnings or low confidence, the UI should display that
-state near the citation.
-
-If confidence is below the configured auto-citation threshold, Athena should
-either avoid using that chunk as evidence or mark the citation as uncertain.
-
-## Agent Loop Specification
-
-### Ownership
-
-Athena must implement its own agent loop. No SDK should own the loop.
-
-The loop must be responsible for:
-
-- building model messages;
-- exposing tool schemas;
-- receiving tool calls;
-- validating tool call inputs;
-- executing tools;
-- appending tool results;
-- enforcing max steps;
-- streaming partial output;
-- compacting conversation state;
-- validating final citations.
-
-### Model Provider Interface
-
-Model providers must be adapters behind a small interface:
-
-```ts
-export interface ModelProvider {
-  streamResponse(input: {
-    model: string;
-    messages: ModelMessage[];
-    tools: ToolDef[];
-    temperature?: number;
-  }): AsyncIterable<ModelEvent>;
-}
-```
-
-Provider adapters may use vendor SDKs internally if useful, but those SDKs must
-not own Athena's agent loop.
-
-### External Agent Provider
-
-OpenCode should remain the default external-agent provider for MCP setup.
-Athena should also allow `codex` through `agent.provider` and the
-`--agent-provider` CLI option. This setting is client-specific setup guidance
-only; it must not replace Athena's loop, weaken citation validation, or expose
-additional MCP tools.
-
-### Step Limit
-
-The default max step count should be 6.
-
-If the model exceeds the limit, Athena must stop the turn and return a clear
-message that the answer could not be completed within the tool budget.
-
-### Citation Validation
-
-Before finalizing an answer, Athena must validate that cited claims reference
-retrieved source ranges or PDF regions.
-
-The validator must reject:
-
-- citations pointing to unknown paths;
-- line ranges outside file bounds;
-- PDF page numbers outside the document;
-- PDF regions missing source-map backing when source-map backing is required;
-- citations to chunks that were not retrieved or read during the turn.
-
-## Agent Tools
-
-### `searchDocs`
-
-Hybrid search over indexed documentation.
-
-Input:
-
-```json
-{
-  "query": "string",
-  "filters": {
-    "pathPrefix": "string",
-    "kind": "markdown | pdf | html | text"
-  },
-  "limit": 10
-}
-```
-
-Output:
-
-```json
-{
-  "results": [
-    {
-      "chunkId": 123,
-      "path": "docs/api.md",
-      "headingPath": "Authentication > Tokens",
-      "snippet": "...",
-      "score": 0.82,
-      "locator": { "startLine": 10, "endLine": 22 }
-    }
-  ]
-}
-```
-
-### `readSourceRange`
-
-Read an exact text-like source range.
-
-Input:
-
-```json
-{
-  "path": "docs/api.md",
-  "startLine": 10,
-  "endLine": 40
-}
-```
-
-### `readPdfSource`
-
-Read converted PDF content and source-map metadata.
-
-Input:
-
-```json
-{
-  "path": "manual.pdf",
-  "page": 17,
-  "regionId": "..."
-}
-```
-
-### `grepDocs`
-
-Exact string or regex search.
-
-Input:
-
-```json
-{
-  "pattern": "AUTH_TOKEN",
-  "caseSensitive": false,
-  "limit": 20
-}
-```
-
-### `listDocs`
-
-Inspect indexed sources.
-
-Input:
-
-```json
-{
-  "pathPrefix": "docs/",
-  "kind": "pdf"
-}
-```
-
-### `relatedChunks`
-
-Fetch neighboring chunks.
-
-Input:
-
-```json
-{
-  "chunkId": 123,
-  "before": 1,
-  "after": 2
-}
-```
-
-## Retrieval Specification
-
-Athena must support hybrid retrieval:
-
-1. lexical search with SQLite FTS5;
-2. semantic vector search;
-3. exact grep for identifiers;
-4. result merging and deduplication;
-5. optional reranking;
-6. neighboring context expansion;
-7. warning/confidence-aware PDF scoring.
-
-Ranking should boost:
-
-- exact identifier matches;
-- heading matches;
-- source title matches;
-- recently read related chunks;
-- high-confidence PDF regions.
-
-Ranking should penalize:
-
-- low-confidence PDF regions;
-- chunks with parser warnings;
-- overly large generic chunks;
-- stale chunks from superseded parser/index versions.
-
-## Embeddings Specification
-
-### Default Provider
-
-The default embedding provider should be local:
-
-```yaml
-provider: transformers-js
-model: mixedbread-ai/mxbai-embed-xsmall-v1
-dimensions: 384
-pooling: mean
-normalize: true
-```
-
-### Optional Providers
-
-Athena should support provider adapters for:
-
-- Ollama;
-- OpenAI;
-- Voyage;
-- Cohere;
-- Gemini.
-
-Remote providers must require explicit user opt-in before document text is sent
-over the network.
-
-### Cache Keys
-
-Embedding cache keys must include:
-
-- chunk text hash;
-- provider name;
-- model name;
-- dimensions;
-- pooling;
-- normalization;
-- quantization settings;
-- model revision or digest when available;
-- chunking strategy version.
-
-## Storage Specification
-
-Athena must store index state in `.athena/index.sqlite`.
-
-Required tables:
-
-- `sources`;
-- `documents`;
-- `chunks`;
-- `chunks_fts`;
-- `embeddings`;
-- `runs`;
-- `parser_artifacts`;
-- `conversation_turns`;
-- `answer_citations`.
-
-The schema must be migration-based and versioned.
-
-Source records must include:
-
-- path;
-- file kind;
-- hash;
-- file size;
-- mtime;
-- parser name;
-- parser version;
-- parser options hash;
-- indexed timestamp.
-
-Chunk records must include:
-
-- source ID;
-- document ID;
-- ordinal;
-- heading path;
-- text;
-- normalized text;
-- token estimate;
-- locator JSON;
-- extraction confidence;
-- warning codes.
-
-## Chat UI Specification
-
-The local UI must include:
-
-- chat panel;
-- source citation list;
-- source preview panel;
-- retrieval trace drawer;
-- index status indicator;
-- parser warning badges;
-- source filter controls.
-
-Citation interactions:
-
-- text citations open exact line ranges;
-- PDF citations open generated Markdown and PDF page-region metadata;
-- uncertain citations show warning/confidence state;
-- copied answers preserve citation labels.
-
-## Follow-Up Memory Specification
-
-Athena should persist compact conversation memory:
-
-- user question;
-- final answer summary;
-- citations;
-- timestamp;
-- optional user-pinned sources.
-
-Athena should not replay full intermediate tool traces into future model context
-unless the user asks for debug/audit details.
-
-## Security and Privacy
-
-Defaults:
-
-- bind the UI to `127.0.0.1`;
-- store indexes locally;
-- use local embeddings;
-- avoid remote document upload;
-- avoid automatic external agent execution.
-
-Remote model or embedding providers must require explicit configuration.
-
-MCP mode must be read-only by default.
-
-## MVP Acceptance Criteria
-
-### MVP-1: Markdown RAG
-
-Done when:
-
-- `athena index` indexes Markdown files;
-- unchanged files are skipped on the second run;
-- `athena serve` starts a local UI;
-- a question can be answered from Markdown;
-- answer citations open exact line ranges.
-
-### MVP-2: Local Embeddings
-
-Done when:
-
-- default local embeddings are generated;
-- vectors are cached;
-- lexical and vector search are merged;
-- changing the embedding model invalidates only affected vectors.
-
-### MVP-3: PDF RAG
-
-Done when:
-
-- `pdf-2-llm` is installed from npm;
-- PDFs are converted through `pdf-2-llm/node`;
-- generated Markdown and source maps are stored;
-- PDF citations include page and region metadata;
-- low-confidence regions are visible in the UI.
-
-### MVP-4: Agent Loop
-
-Done when:
-
-- the custom loop can call tools for up to a configured max step count;
-- no agent SDK controls tool execution;
-- invalid tool inputs are rejected;
-- final answer citation validation runs every turn;
-- unsupported answers are refused or marked as unsupported.
-
-### MVP-5: Evaluation Harness
-
-Done when:
-
-- parser/index tests run locally;
-- retrieval tests compare FTS-only, vector-only, and hybrid results;
-- answer tests validate citation requirements;
-- PDF source-map tests verify page and region locators.
-
-## Open Questions
-
-- Which model provider should be the first chat-generation adapter?
-- Should the initial UI be a small custom app or a minimal server-rendered page?
-- Should vector search use `sqlite-vec` immediately or begin with a simple
-  in-process vector scan for MVP scale?
-- What exact `pdf-2-llm` output fields should be considered stable public API?
-- Should MCP mode launch from the same index server or as a separate process?