@alextis59/athena 1.0.0 → 1.1.0

package/docs/QUALITY.md

@@ -0,0 +1,137 @@
+# Quality
+
+Use the narrowest validation that proves the change, then expand when the
+change touches shared behavior, package output, retrieval quality, security
+boundaries, or user-facing workflows.
+
+## Fast Checks
+
+For documentation-only changes:
+
+```bash
+npm run format:check
+```
+
+For TypeScript source changes, start with the nearest targeted Node test when
+possible:
+
+```bash
+node --import tsx --test test/<area>.test.ts
+```
+
+Then run the relevant package checks from the sections below.
+
+## Standard Source Change Ladder
+
+Run this for most runtime changes:
+
+```bash
+npm run format:check
+npm run lint
+npm run typecheck
+npm test
+npm run build
+```
+
+`npm run check` combines the first four commands:
+
+```bash
+npm run check
+```
+
+Prefer the expanded form when you need to show the exact failing stage.
+
+## Area-Specific Tests
+
+- CLI behavior: `node --import tsx --test test/cli.test.ts`
+- Config loading and remote opt-ins: `node --import tsx --test test/config.test.ts`
+- Storage and migrations: `node --import tsx --test test/storage.test.ts`
+- Indexing and run metadata: `node --import tsx --test test/indexer.test.ts`
+- Parsers and PDF artifacts: `node --import tsx --test test/parsers.test.ts`
+- Embedding providers and cache keys: `node --import tsx --test test/embeddings.test.ts`
+- Retrieval tools and ranking: `node --import tsx --test test/retrieval.test.ts`
+- Agent loop and provider adapters: `node --import tsx --test test/agent-loop.test.ts test/agent-provider.test.ts`
+- Citation policy: `node --import tsx --test test/agent-citations.test.ts`
+- HTTP server and UI API contract: `node --import tsx --test test/server.test.ts test/ui.test.ts`
+- MCP protocol and tool exposure: `node --import tsx --test test/mcp.test.ts`
+- Corpus manifest and eval plan: `node --import tsx --test test/corpus.test.ts test/eval.test.ts`
+
+When changing cross-cutting types, exports, or shared contracts, run the full
+standard ladder after targeted tests pass.
+
+## Retrieval, Parser, And Citation Quality
+
+For changes that affect discovery, parsing, chunking, embeddings, retrieval,
+ranking, PDF handling, source locators, agent tool use, or citations, include:
+
+```bash
+npm run corpus:health
+npm run eval
+```
+
+`npm run eval` runs offline validation against the committed corpus manifest and
+ground truth. If corpus cache files are missing, use:
+
+```bash
+npm run corpus:retrieve
+npm run corpus:health
+```
+
+Only update corpus manifest hashes or sizes after reviewing the changed
+downloaded artifact and documenting why the refresh is acceptable.
+
+## Manual CLI Smoke
+
+Use a temporary project root when validating real CLI behavior:
+
+```bash
+tmpdir="$(mktemp -d)"
+mkdir -p "$tmpdir/docs"
+printf '# Admin Guide\n\nSet ATHENA_INDEX_BATCH_SIZE=64 for large docsets.\n' > "$tmpdir/docs/admin-guide.md"
+npm run build
+node dist/cli/main.js index --root "$tmpdir" --json
+node dist/cli/main.js doctor --root "$tmpdir"
+```
+
+For `serve`, prefer a test harness or short-lived manual run. The server waits
+for shutdown by default.
+
+## MCP Smoke
+
+For MCP changes, run the targeted MCP tests and use the finite stdio smoke in
+`docs/mcp-integration.md` when the wire format or exposed tools change.
+
+## Package And Publish Readiness
+
+For release or package-entrypoint changes:
+
+```bash
+npm run check:publish
+npm pack --dry-run --json
+```
+
+Then smoke the generated tarball from this checkout:
+
+```bash
+npm pack
+npx --yes --package ./alextis59-athena-1.0.0.tgz athena --help
+```
+
+Also smoke an index command against a temporary docs folder before publishing.
+Use `npm publish --dry-run --json` for final registry packaging validation.
+Remove local `*.tgz` artifacts after the smoke test.
+
+The npm package name is `@alextis59/athena`; the installed command must remain
+`athena`.
+
+## What To Record
+
+In final notes, include:
+
+- commands run;
+- pass/fail outcome;
+- any skipped checks and why;
+- manual smoke details when automation does not cover the behavior.
+
+Do not mark a plan item or debt item complete unless the matching validation has
+actually passed.

package/docs/SECURITY.md

@@ -0,0 +1,140 @@
+# Security
+
+Athena is a local developer tool for indexing and querying project
+documentation. Its default security posture is local-first operation, explicit
+remote document-text opt-in, read-only retrieval integrations, and no public
+service exposure.
+
+## Trust Model
+
+- Trusted operator: the local user running `athena`.
+- Trusted codebase: this package and installed runtime dependencies.
+- Trusted data root: the `--root` directory selected by the user.
+- Untrusted inputs: documentation content, user config files in projects being
+  indexed, model output, MCP client requests, HTTP query parameters, and remote
+  provider responses.
+
+Athena does not sandbox arbitrary project config execution. `athena.config.ts`
+and `athena.config.mjs` are loaded with `jiti`, so only use config files from
+repositories you are willing to execute locally.
+
+## Local-First Boundaries
+
+By default:
+
+- indexes are written under `.athena/index.sqlite`;
+- generated PDF assets are written under `.athena/blobs/`;
+- the default embedding provider is local Transformers.js with remote model
+  downloads disabled at runtime;
+- chat is disabled until the user configures a provider;
+- the external agent provider defaults to OpenCode and only changes
+  client-specific MCP initialization guidance;
+- the HTTP server binds to `127.0.0.1:4317`.
+
+Do not change defaults so that document text leaves the machine. Remote
+embedding providers must require `embeddings.allowRemoteDocumentText: true`.
+Remote chat providers must require `chat.allowRemoteDocumentText: true`.
+
+## Secrets
+
+Remote providers use environment variables:
+
+- `OPENAI_API_KEY`
+- `VOYAGE_API_KEY`
+- `COHERE_API_KEY`
+- `GEMINI_API_KEY`
+- `ANTHROPIC_API_KEY` for configured chat validation
+
+Do not log API key values, persist them in `.athena/`, commit them, or include
+them in doctor output. The current doctor command reports provider/model/cache
+state, not secret values.
+
+## Network
+
+Expected network surfaces:
+
+- optional remote embedding APIs when explicitly configured and allowed;
+- optional Ollama embedding/chat requests to `OLLAMA_HOST` or local defaults;
+- optional corpus retrieval through `npm run corpus:retrieve`;
+- npm/package operations run manually by maintainers.
+
+The local web server has no authentication or authorization layer. Keep it on
+loopback for normal use. If `--host` or config binds to a non-loopback address,
+the operator is responsible for network controls until an authenticated server
+mode exists.
+
+## Filesystem
+
+Indexing reads files discovered under the configured root using supported
+documentation include/exclude patterns. Default exclusions cover `.git/`,
+`node_modules/`, `.athena/`, `dist/`, `build/`, and `coverage/`.
+
+Important filesystem constraints:
+
+- discovery does not follow symbolic links;
+- retrieval source reads require indexed repository-relative paths;
+- citation validation rejects absolute paths and paths containing `..`;
+- server source preview rejects absolute paths, `..`, and paths outside the
+  selected root;
+- PDF sidecar asset reads are restricted to `.athena/blobs/pdf-assets/`;
+- downloaded validation corpus artifacts belong in ignored
+  `test/corpus/cache/`.
+
+Do not commit `.athena/`, `dist/`, `*.tgz`, model caches, `.env*`, or
+downloaded corpus cache files.
+
+## MCP
+
+`athena mcp` exposes read-only retrieval tools:
+
+- `searchDocs`
+- `readSourceRange`
+- `readPdfSource`
+- `grepDocs`
+- `listDocs`
+- `relatedChunks`
+
+It does not expose write, edit, delete, indexing, shell, chat, or package
+management tools. Keep MCP read-only unless a future change includes explicit
+product requirements, authentication/authorization design, and tests.
+
+`agent.provider` and `--agent-provider` may select OpenCode or Codex setup
+guidance, but they must not change the read-only tool set or let an external
+agent own Athena's model loop.
+
+## Model Output And Citations
+
+Treat model output as untrusted. Athena validates citations before accepting an
+answer:
+
+- source paths must be indexed and repository-relative;
+- text citation line ranges must exist;
+- PDF citations must reference PDF page labels;
+- cited locators must have been read during the turn;
+- substantive answers require citations.
+
+Do not weaken citation validation to make a model answer pass. Fix retrieval,
+tool use, or the answer policy instead.
+
+## Parser And Document Content
+
+Markdown, MDX, HTML, RST, AsciiDoc, text, and PDF inputs are parsed as
+documentation content. Embedded code blocks, MDX JSX, script tags, and style
+tags must not execute as part of parsing. PDF parsing is bounded by configured
+timeout and wrapped in user-facing errors.
+
+When changing parsers, preserve warning behavior and avoid turning malformed or
+hostile documents into process crashes.
+
+## Practical Review Checklist
+
+Before merging changes that affect config, providers, server routes, MCP,
+filesystem reads, parsing, or citations, verify:
+
+- remote document text still requires explicit opt-in;
+- no secret value is logged, persisted, or printed;
+- source and asset path traversal checks still exist;
+- MCP remains read-only;
+- the server is not documented as safe for public exposure;
+- citation validation still rejects unsupported or unread citations;
+- relevant tests from `docs/QUALITY.md` pass.

package/docs/agentic-rag-specifications.md

@@ -183,6 +183,9 @@ export default defineConfig({
     pooling: "mean",
     normalize: true
   },
+  agent: {
+    provider: "opencode"
+  },
   pdf: {
     parser: "pdf-2-llm",
     pageAnchors: true,
@@ -317,6 +320,14 @@ export interface ModelProvider {
 Provider adapters may use vendor SDKs internally if useful, but those SDKs must
 not own Athena's agent loop.
 
+### External Agent Provider
+
+OpenCode should remain the default external-agent provider for MCP setup.
+Athena should also allow `codex` through `agent.provider` and the
+`--agent-provider` CLI option. This setting is client-specific setup guidance
+only; it must not replace Athena's loop, weaken citation validation, or expose
+additional MCP tools.
+
 ### Step Limit
 
 The default max step count should be 6.

package/docs/exec-plans/template.md

@@ -0,0 +1,67 @@
+# Execution Plan: <title>
+
+Date: <yyyy-mm-dd>
+
+## Goal
+
+Describe the user-visible outcome and the repository behavior that will change.
+
+## Scope
+
+In scope:
+
+- <files, modules, commands, docs, or workflows>
+
+Out of scope:
+
+- <explicitly deferred or preserved behavior>
+
+## Current Behavior
+
+Summarize what the code and docs do now. Link the relevant files, tests, and
+commands inspected before editing.
+
+## Constraints And Invariants
+
+- Preserve the `athena` binary name and `@alextis59/athena` package identity.
+- Preserve local-first defaults and explicit remote document-text opt-ins.
+- Preserve Athena-owned agent loop and citation validation unless this plan is
+  explicitly about changing them.
+- Preserve `pdf-2-llm` as the PDF ingestion path unless this plan is explicitly
+  about parser architecture.
+- Keep MCP read-only unless this plan includes a security design for writes.
+
+Add task-specific constraints here.
+
+## Plan
+
+1. Inspect the relevant code, docs, tests, and package scripts.
+2. Make the smallest implementation or documentation change that satisfies the
+   goal.
+3. Update adjacent docs, tests, fixtures, or debt notes when behavior changes.
+4. Run the validation below.
+5. Record results and any follow-up debt.
+
+## Validation
+
+Commands to run:
+
+```bash
+<command>
+```
+
+Expected outcomes:
+
+- <observable pass condition>
+
+If validation is skipped, record the reason and residual risk.
+
+## Rollback
+
+Describe how to revert the change or disable the behavior if validation fails
+after merge.
+
+## Notes
+
+Capture decisions, tradeoffs, and follow-up items that should survive the
+implementation turn.

package/docs/implementation-plan.md

@@ -1279,6 +1279,14 @@ Milestone 11 external agent documentation proof, 2026-07-03:
   returned 4 JSON-RPC responses, 6 tools, `docs/admin-guide.md` as the top
   search hit, and 1 read trace locator.
 
+Milestone 11 external agent provider option proof, 2026-07-05:
+
+- Added `agent.provider` with `opencode` as the default and `codex` as the
+  alternate provider.
+- Added `athena mcp --agent-provider <opencode|codex>` for one-off MCP setup.
+- Updated OpenCode and Codex MCP setup docs while preserving the same read-only
+  tool exposure for both providers.
+
 ## Release Readiness Checklist
 
 Use this before publishing a public package or tagging a release.

package/docs/index.md

@@ -0,0 +1,61 @@
+# Documentation Index
+
+Start with the root files, then use the task-specific docs below.
+
+## Agent Entry Points
+
+- [../AGENTS.md](../AGENTS.md): repository instructions for coding agents.
+- [../ARCHITECTURE.md](../ARCHITECTURE.md): current package, module, runtime,
+  data-flow, and invariant map.
+- [../README.md](../README.md): user-facing install, command, config, MCP, and
+  development quickstart.
+
+## Product And Implementation
+
+- [agentic-rag-specifications.md](agentic-rag-specifications.md): product and
+  engineering requirements for Athena's local-first documentation RAG behavior.
+- [agentic-rag-study.md](agentic-rag-study.md): source study behind the
+  architecture choices.
+- [implementation-plan.md](implementation-plan.md): milestone checklist and
+  historical proof notes for implementation work.
+- [mcp-integration.md](mcp-integration.md): OpenCode, Codex, and generic MCP
+  setup, exposed read-only tools, and manual stdio smoke test.
+- [validation-corpus.md](validation-corpus.md): corpus layout, licensing,
+  retrieval, cache, and coverage rules.
+
+## Harness Operations
+
+- [SECURITY.md](SECURITY.md): security model, local-first boundaries, secrets,
+  network, filesystem, MCP, and server exposure notes.
+- [QUALITY.md](QUALITY.md): validation ladder for docs, source changes,
+  retrieval/parser changes, corpus changes, package release checks, and manual
+  smoke tests.
+- [tech-debt-tracker.md](tech-debt-tracker.md): practical deferred work that is
+  visible from the current code and docs.
+- [exec-plans/template.md](exec-plans/template.md): reusable plan template for
+  larger changes.
+
+## Decisions
+
+- [adr/0001-sqlite-driver.md](adr/0001-sqlite-driver.md): why Athena uses
+  `better-sqlite3` for the local SQLite layer.
+
+## Source Map
+
+- `src/cli/`: command parsing and dispatch.
+- `src/config/`: default config, config loading, validation, and
+  `defineConfig()`.
+- `src/storage/`: `.athena/` paths, migrations, schema checks, and
+  transactions.
+- `src/indexer/`: discovery, hashing, parsing, chunking, embedding, and index
+  writes.
+- `src/parsers/`: text-like parsers, PDF parsing, chunking, and normalization.
+- `src/embeddings/`: local, Ollama, and remote embedding providers.
+- `src/retrieval/`: read-only retrieval tools and trace capture.
+- `src/agent/`: Athena-owned agent loop, model providers, tools, citations, and
+  turn persistence.
+- `src/server/`: local HTTP UI/API server.
+- `src/ui/`: static web UI.
+- `src/mcp/`: stdio MCP server.
+- `scripts/`: corpus and evaluation commands.
+- `test/`: Node test suites and validation corpus fixtures.

package/docs/mcp-integration.md

@@ -17,6 +17,31 @@ node dist/cli/main.js index --root /absolute/path/to/project
 For an installed package, replace `node dist/cli/main.js` with `athena` or
 `npx @alextis59/athena`.
 
+## Agent Provider Option
+
+OpenCode is the default external-agent provider. Use `--agent-provider codex`
+when the MCP server is configured for Codex:
+
+```sh
+athena mcp --root /absolute/path/to/project --agent-provider codex
+```
+
+The same choice can be made durable in `athena.config.ts`,
+`athena.config.mjs`, or `athena.config.json`:
+
+```ts
+import { defineConfig } from "@alextis59/athena";
+
+export default defineConfig({
+  agent: {
+    provider: "codex"
+  }
+});
+```
+
+The option changes client-specific initialization guidance only. The MCP server
+continues to expose the same read-only retrieval tools for every provider.
+
 ## OpenCode
 
 OpenCode configures MCP servers under the `mcp` key. A local MCP entry uses
@@ -66,6 +91,54 @@ Installed package example:
 After saving the config, run `opencode mcp list` to check that OpenCode can
 start the server and discover its tools.
 
+## Codex
+
+Codex stores MCP servers in `config.toml`. You can configure Athena in the
+user-level `~/.codex/config.toml` or in a trusted project-scoped
+`.codex/config.toml`.
+
+Installed package example:
+
+```toml
+[mcp_servers.athena]
+command = "npx"
+args = [
+  "--yes",
+  "@alextis59/athena",
+  "mcp",
+  "--root",
+  ".",
+  "--agent-provider",
+  "codex"
+]
+cwd = "/absolute/path/to/project"
+enabled = true
+startup_timeout_sec = 10
+tool_timeout_sec = 60
+```
+
+Development checkout example:
+
+```toml
+[mcp_servers.athena]
+command = "node"
+args = [
+  "/absolute/path/to/athena/dist/cli/main.js",
+  "mcp",
+  "--root",
+  ".",
+  "--agent-provider",
+  "codex"
+]
+cwd = "/absolute/path/to/project"
+enabled = true
+startup_timeout_sec = 10
+tool_timeout_sec = 60
+```
+
+After saving the config, start Codex in the project and use `/mcp` in the
+terminal UI to check that the `athena` server is connected.
+
 ## Exposed Tools
 
 Athena exposes only read-only retrieval tools:

package/docs/tech-debt-tracker.md

@@ -0,0 +1,21 @@
+# Tech Debt Tracker
+
+Track deferred work that is visible from the current implementation. Keep this
+file practical: each item should have a reason, a likely validation path, and a
+status.
+
+| Item | Context | Validation | Owner | Status |
+| --- | --- | --- | --- | --- |
+| Align chat provider docs and adapters | Config validation recognizes `openai`, `anthropic`, and `gemini` chat providers, but `createConfiguredModelProvider()` currently implements Ollama only after validation. Either add adapters or keep user docs explicit that non-Ollama chat adapters are not ready. | Provider tests plus `athena ask` smoke for each supported adapter. | TBD | Open |
+| Decide authenticated server posture before non-loopback use | The HTTP server is intentionally local and unauthenticated today. Binding `--host` to a non-loopback address should remain an operator-controlled advanced use until there is an authenticated mode. | Server tests for auth behavior if added; security doc update. | TBD | Open |
+| Evaluate query embedding parity for remote/provider-specific indexes | Indexed vectors can come from provider-specific embeddings. Retrieval currently builds query vectors inside the retrieval layer rather than routing the query through the configured embedding provider. Confirm whether this is acceptable for release quality or add provider-aware query embedding. | Retrieval/eval tests comparing hybrid, FTS-only, vector-only, and provider-backed query behavior. | TBD | Open |
+| Harden package smoke automation | Package readiness currently relies on manual tarball and `npx --package` smoke commands documented in `docs/QUALITY.md`. A script could reduce release mistakes while preserving the same checks. | New script plus tarball CLI smoke in a temp project. | TBD | Open |
+| Expand parser warning summaries | Doctor reports total parser warnings, while PDF warning detail is available in parser artifacts and retrieval responses. A future doctor view could summarize warning codes by source. | Doctor tests and fixture with repeated PDF warnings. | TBD | Open |
+
+## Closing Criteria
+
+Close an item only after:
+
+- the implementation or documentation change lands;
+- the validation in the row passes;
+- related docs are updated when behavior or policy changes.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@alextis59/athena",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Local-first agentic documentation RAG command.",
   "type": "module",
   "private": false,