@alextis59/athena 1.0.0

package/dist/ui/static-app.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"static-app.js","sourceRoot":"","sources":["../../src/ui/static-app.ts"],"names":[],"mappings":"AAAA,MAAM,UAAU,aAAa;IAC3B,OAAO;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;QA2tBD,CAAC;AACT,CAAC"}

package/docs/adr/0001-sqlite-driver.md

@@ -0,0 +1,23 @@
+# ADR 0001: SQLite Driver
+
+Date: 2026-07-03
+
+## Decision
+
+Athena uses `better-sqlite3` for the initial local SQLite storage layer.
+
+## Rationale
+
+- Athena's index updates and retrieval reads are local, short-lived CLI/server
+  operations where synchronous transactions are simple and deterministic.
+- `better-sqlite3` provides mature SQLite and FTS5 support without the
+  `node:sqlite` ExperimentalWarning emitted by the current Node 22 runtime.
+- The storage boundary is isolated in `src/storage/`, so a future migration to a
+  different SQLite binding or vector extension can be handled behind the same
+  migration and transaction helpers.
+
+## Consequences
+
+- `better-sqlite3` is a runtime dependency.
+- Tests can inspect schema, indexes, and rollback behavior without a separate
+  database service.

package/docs/agentic-rag-specifications.md

@@ -0,0 +1,659 @@
+# Agentic RAG Tool Specifications
+
+Date: 2026-07-03
+
+Related study: [Agentic RAG Tool Design Study](./agentic-rag-study.md)
+
+## Purpose
+
+Athena is a local-first documentation RAG tool. It indexes documentation in the
+current folder, launches a local chat UI, and answers questions with exact
+citations into the original sources.
+
+This specification converts the study into concrete product and engineering
+requirements.
+
+## Naming
+
+- Project name: `athena`.
+- npm package name: `@alextis59/athena`.
+- CLI binary/command name: `athena`.
+- Package-run form: `npx @alextis59/athena`.
+- Local cache directory: `.athena/`.
+
+All product documentation and command examples should treat `athena` as the
+canonical project and command name. `npx @alextis59/athena` is only the
+no-install npm invocation form.
+
+## Final Architecture Decisions
+
+1. Athena owns the agentic loop.
+   No agent SDK handles tool selection, tool execution, step control, memory
+   policy, streaming, or citation validation.
+
+2. `pdf-2-llm` is the canonical PDF parser.
+   PDF ingestion uses the published npm package `pdf-2-llm`, not a local
+   workspace-only dependency.
+
+3. Local-first is the default.
+   Indexes, embeddings, and chat UI should run locally unless the user
+   explicitly opts into remote model or embedding providers.
+
+4. Citations are first-class data.
+   Every substantive answer must be grounded in source ranges or PDF page
+   regions that the UI can open.
+
+5. External agents are integrations.
+   OpenCode, MCP clients, LangChain, LlamaIndex, Vercel AI SDK, and similar
+   tools may integrate with Athena, but none of them should own Athena's core
+   agent behavior.
+
+## Functional Requirements
+
+### FR-1: Single Command Startup
+
+Athena must support a command that can be run inside any documentation folder:
+
+```bash
+athena
+```
+
+For ad hoc execution without installing the binary first, users may run
+`npx @alextis59/athena`.
+
+The command must:
+
+- discover supported documentation files;
+- create or update a local `.athena/index.sqlite` cache;
+- skip unchanged files;
+- start a local web UI;
+- expose a chat interface connected to the local index.
+
+### FR-2: Index-Only Mode
+
+Athena must support indexing without starting the UI:
+
+```bash
+athena index
+```
+
+This command must exit non-zero when parsing, database migration, or embedding
+generation fails.
+
+### FR-3: Serve Existing Index
+
+Athena must support serving a previously built index:
+
+```bash
+athena serve
+```
+
+If the index is missing, the command should either fail with a clear message or
+offer an explicit `--build` mode.
+
+### FR-4: One-Shot Ask Mode
+
+Athena should support terminal questions:
+
+```bash
+athena ask "How do I configure OAuth?"
+```
+
+The answer must include citations in terminal-readable form.
+
+### FR-5: Doctor Mode
+
+Athena should expose a diagnostic command:
+
+```bash
+athena doctor
+```
+
+It should report:
+
+- database status;
+- schema version;
+- indexed source count;
+- embedding provider status;
+- `pdf-2-llm` package availability;
+- parser warnings summary;
+- model cache status;
+- UI port availability.
+
+### FR-6: MCP Server Mode
+
+Athena should expose an optional MCP server:
+
+```bash
+athena mcp
+```
+
+The MCP server must expose citation-aware retrieval tools only. It must not
+allow external agents to bypass Athena's source validation.
+
+## Supported Inputs
+
+### Initial File Types
+
+Athena must support:
+
+- Markdown: `.md`, `.mdx`;
+- text: `.txt`;
+- reStructuredText: `.rst`;
+- AsciiDoc: `.adoc`;
+- HTML: `.html`, `.htm`;
+- PDF: `.pdf`;
+- extensionless documentation files such as `README`, `CHANGELOG`,
+  `CONTRIBUTING`, and `LICENSE`.
+
+### Default Exclusions
+
+Athena must exclude:
+
+- `.git/**`;
+- `node_modules/**`;
+- `.athena/**`;
+- `dist/**`;
+- `build/**`;
+- `coverage/**`;
+- binary files not explicitly supported.
+
+## Configuration
+
+Athena should load optional config from:
+
+- `athena.config.ts`;
+- `athena.config.mjs`;
+- `athena.config.json`.
+
+Example:
+
+```ts
+import { defineConfig } from "@alextis59/athena";
+
+export default defineConfig({
+  index: {
+    include: ["**/*.md", "**/*.mdx", "**/*.pdf", "**/*.html"],
+    exclude: ["node_modules/**", "dist/**", ".git/**"]
+  },
+  embeddings: {
+    provider: "transformers-js",
+    model: "mixedbread-ai/mxbai-embed-xsmall-v1",
+    dimensions: 384,
+    pooling: "mean",
+    normalize: true
+  },
+  pdf: {
+    parser: "pdf-2-llm",
+    pageAnchors: true,
+    csvSidecars: true,
+    timeoutMs: 30_000,
+    minConfidenceForAutoCitation: 0.65
+  },
+  server: {
+    host: "127.0.0.1",
+    port: 4317
+  }
+});
+```
+
+## PDF Ingestion Specification
+
+### Package
+
+Athena must use the public npm package:
+
+```bash
+npm install pdf-2-llm
+```
+
+The Node parser import should use:
+
+```ts
+import { convertPdfToMarkdown } from "pdf-2-llm/node";
+```
+
+### PDF Parser Output
+
+Athena must preserve these `pdf-2-llm` outputs when available:
+
+- Markdown;
+- source-map entries;
+- page regions;
+- document IR;
+- table sidecars;
+- extracted assets;
+- warning codes;
+- diagnostics;
+- confidence scores.
+
+### PDF Storage Requirements
+
+For every parsed PDF, Athena must store:
+
+- original file path;
+- source hash;
+- parser package version;
+- parser options hash;
+- generated Markdown;
+- source-map metadata;
+- document IR metadata;
+- warning summary;
+- confidence summary;
+- sidecar asset references.
+
+### PDF Citation Requirements
+
+PDF citations must include:
+
+- source file path;
+- page number;
+- page region coordinates when available;
+- generated Markdown offset or line range;
+- confidence score when available;
+- warning codes that affect the cited region.
+
+Example:
+
+```json
+{
+  "kind": "pdf",
+  "path": "manual.pdf",
+  "page": 17,
+  "regions": [
+    { "x": 72, "y": 144, "width": 440, "height": 88 }
+  ],
+  "markdownOffsetStart": 1204,
+  "markdownOffsetEnd": 1422,
+  "confidence": 0.91,
+  "warningCodes": []
+}
+```
+
+### PDF Warning Behavior
+
+Athena must not silently hide extraction uncertainty.
+
+If a cited PDF region has warnings or low confidence, the UI should display that
+state near the citation.
+
+If confidence is below the configured auto-citation threshold, Athena should
+either avoid using that chunk as evidence or mark the citation as uncertain.
+
+## Agent Loop Specification
+
+### Ownership
+
+Athena must implement its own agent loop. No SDK should own the loop.
+
+The loop must be responsible for:
+
+- building model messages;
+- exposing tool schemas;
+- receiving tool calls;
+- validating tool call inputs;
+- executing tools;
+- appending tool results;
+- enforcing max steps;
+- streaming partial output;
+- compacting conversation state;
+- validating final citations.
+
+### Model Provider Interface
+
+Model providers must be adapters behind a small interface:
+
+```ts
+export interface ModelProvider {
+  streamResponse(input: {
+    model: string;
+    messages: ModelMessage[];
+    tools: ToolDef[];
+    temperature?: number;
+  }): AsyncIterable<ModelEvent>;
+}
+```
+
+Provider adapters may use vendor SDKs internally if useful, but those SDKs must
+not own Athena's agent loop.
+
+### Step Limit
+
+The default max step count should be 6.
+
+If the model exceeds the limit, Athena must stop the turn and return a clear
+message that the answer could not be completed within the tool budget.
+
+### Citation Validation
+
+Before finalizing an answer, Athena must validate that cited claims reference
+retrieved source ranges or PDF regions.
+
+The validator must reject:
+
+- citations pointing to unknown paths;
+- line ranges outside file bounds;
+- PDF page numbers outside the document;
+- PDF regions missing source-map backing when source-map backing is required;
+- citations to chunks that were not retrieved or read during the turn.
+
+## Agent Tools
+
+### `searchDocs`
+
+Hybrid search over indexed documentation.
+
+Input:
+
+```json
+{
+  "query": "string",
+  "filters": {
+    "pathPrefix": "string",
+    "kind": "markdown | pdf | html | text"
+  },
+  "limit": 10
+}
+```
+
+Output:
+
+```json
+{
+  "results": [
+    {
+      "chunkId": 123,
+      "path": "docs/api.md",
+      "headingPath": "Authentication > Tokens",
+      "snippet": "...",
+      "score": 0.82,
+      "locator": { "startLine": 10, "endLine": 22 }
+    }
+  ]
+}
+```
+
+### `readSourceRange`
+
+Read an exact text-like source range.
+
+Input:
+
+```json
+{
+  "path": "docs/api.md",
+  "startLine": 10,
+  "endLine": 40
+}
+```
+
+### `readPdfSource`
+
+Read converted PDF content and source-map metadata.
+
+Input:
+
+```json
+{
+  "path": "manual.pdf",
+  "page": 17,
+  "regionId": "..."
+}
+```
+
+### `grepDocs`
+
+Exact string or regex search.
+
+Input:
+
+```json
+{
+  "pattern": "AUTH_TOKEN",
+  "caseSensitive": false,
+  "limit": 20
+}
+```
+
+### `listDocs`
+
+Inspect indexed sources.
+
+Input:
+
+```json
+{
+  "pathPrefix": "docs/",
+  "kind": "pdf"
+}
+```
+
+### `relatedChunks`
+
+Fetch neighboring chunks.
+
+Input:
+
+```json
+{
+  "chunkId": 123,
+  "before": 1,
+  "after": 2
+}
+```
+
+## Retrieval Specification
+
+Athena must support hybrid retrieval:
+
+1. lexical search with SQLite FTS5;
+2. semantic vector search;
+3. exact grep for identifiers;
+4. result merging and deduplication;
+5. optional reranking;
+6. neighboring context expansion;
+7. warning/confidence-aware PDF scoring.
+
+Ranking should boost:
+
+- exact identifier matches;
+- heading matches;
+- source title matches;
+- recently read related chunks;
+- high-confidence PDF regions.
+
+Ranking should penalize:
+
+- low-confidence PDF regions;
+- chunks with parser warnings;
+- overly large generic chunks;
+- stale chunks from superseded parser/index versions.
+
+## Embeddings Specification
+
+### Default Provider
+
+The default embedding provider should be local:
+
+```yaml
+provider: transformers-js
+model: mixedbread-ai/mxbai-embed-xsmall-v1
+dimensions: 384
+pooling: mean
+normalize: true
+```
+
+### Optional Providers
+
+Athena should support provider adapters for:
+
+- Ollama;
+- OpenAI;
+- Voyage;
+- Cohere;
+- Gemini.
+
+Remote providers must require explicit user opt-in before document text is sent
+over the network.
+
+### Cache Keys
+
+Embedding cache keys must include:
+
+- chunk text hash;
+- provider name;
+- model name;
+- dimensions;
+- pooling;
+- normalization;
+- quantization settings;
+- model revision or digest when available;
+- chunking strategy version.
+
+## Storage Specification
+
+Athena must store index state in `.athena/index.sqlite`.
+
+Required tables:
+
+- `sources`;
+- `documents`;
+- `chunks`;
+- `chunks_fts`;
+- `embeddings`;
+- `runs`;
+- `parser_artifacts`;
+- `conversation_turns`;
+- `answer_citations`.
+
+The schema must be migration-based and versioned.
+
+Source records must include:
+
+- path;
+- file kind;
+- hash;
+- file size;
+- mtime;
+- parser name;
+- parser version;
+- parser options hash;
+- indexed timestamp.
+
+Chunk records must include:
+
+- source ID;
+- document ID;
+- ordinal;
+- heading path;
+- text;
+- normalized text;
+- token estimate;
+- locator JSON;
+- extraction confidence;
+- warning codes.
+
+## Chat UI Specification
+
+The local UI must include:
+
+- chat panel;
+- source citation list;
+- source preview panel;
+- retrieval trace drawer;
+- index status indicator;
+- parser warning badges;
+- source filter controls.
+
+Citation interactions:
+
+- text citations open exact line ranges;
+- PDF citations open generated Markdown and PDF page-region metadata;
+- uncertain citations show warning/confidence state;
+- copied answers preserve citation labels.
+
+## Follow-Up Memory Specification
+
+Athena should persist compact conversation memory:
+
+- user question;
+- final answer summary;
+- citations;
+- timestamp;
+- optional user-pinned sources.
+
+Athena should not replay full intermediate tool traces into future model context
+unless the user asks for debug/audit details.
+
+## Security and Privacy
+
+Defaults:
+
+- bind the UI to `127.0.0.1`;
+- store indexes locally;
+- use local embeddings;
+- avoid remote document upload;
+- avoid automatic external agent execution.
+
+Remote model or embedding providers must require explicit configuration.
+
+MCP mode must be read-only by default.
+
+## MVP Acceptance Criteria
+
+### MVP-1: Markdown RAG
+
+Done when:
+
+- `athena index` indexes Markdown files;
+- unchanged files are skipped on the second run;
+- `athena serve` starts a local UI;
+- a question can be answered from Markdown;
+- answer citations open exact line ranges.
+
+### MVP-2: Local Embeddings
+
+Done when:
+
+- default local embeddings are generated;
+- vectors are cached;
+- lexical and vector search are merged;
+- changing the embedding model invalidates only affected vectors.
+
+### MVP-3: PDF RAG
+
+Done when:
+
+- `pdf-2-llm` is installed from npm;
+- PDFs are converted through `pdf-2-llm/node`;
+- generated Markdown and source maps are stored;
+- PDF citations include page and region metadata;
+- low-confidence regions are visible in the UI.
+
+### MVP-4: Agent Loop
+
+Done when:
+
+- the custom loop can call tools for up to a configured max step count;
+- no agent SDK controls tool execution;
+- invalid tool inputs are rejected;
+- final answer citation validation runs every turn;
+- unsupported answers are refused or marked as unsupported.
+
+### MVP-5: Evaluation Harness
+
+Done when:
+
+- parser/index tests run locally;
+- retrieval tests compare FTS-only, vector-only, and hybrid results;
+- answer tests validate citation requirements;
+- PDF source-map tests verify page and region locators.
+
+## Open Questions
+
+- Which model provider should be the first chat-generation adapter?
+- Should the initial UI be a small custom app or a minimal server-rendered page?
+- Should vector search use `sqlite-vec` immediately or begin with a simple
+  in-process vector scan for MVP scale?
+- What exact `pdf-2-llm` output fields should be considered stable public API?
+- Should MCP mode launch from the same index server or as a separate process?