@mfittko/repo-wiki 0.2.1

package/docs/plans/llm-compiler.md

@@ -0,0 +1,160 @@
+# Epic: LLM Compiler
+
+## Summary
+
+Replace the deterministic placeholder summaries in the wiki compiler with LLM-powered synthesis that produces human-quality wiki pages grounded in source cards, documentation cards, targeted code excerpts, and the current state of an existing wiki when back-filling or reconciling.
+
+## Architecture
+
+```mermaid
+flowchart TD
+  SourceCards[Source Cards] --> Budget[Token Budget Assembler]
+  DocCards[Documentation Cards] --> Budget
+  CodeExcerpts[Targeted Code Excerpts] --> Budget
+  ExistingWiki[Existing Wiki Pages] --> Ownership[Ownership + Preserve Section Extraction]
+  Budget --> Context[Assembled Context Window]
+  Ownership --> Context
+  Context --> Prompt[Prompt Template Selection]
+  PageType[Page Archetype] --> Prompt
+  Prompt --> LLM[LLM Provider]
+  LLM --> RawOutput[Raw LLM Output]
+  RawOutput --> Patch[Structured Patch]
+  Patch --> Preserve[Human Section Preservation]
+  Preserve --> Cite[Citation Enforcement]
+  Cite --> Lint[Lint Gate Validation]
+  Lint -->|pass| Page[Final Wiki Page]
+  Lint -->|fail| Retry[Retry / Flag]
+```
+
+```mermaid
+flowchart LR
+  subgraph Page Archetypes
+    Foundation[Foundation Pages]
+    Module[Module Pages]
+    CrossCut[Cross-cutting Pages]
+  end
+  subgraph Prompts
+    P1[home.md]
+    P2[architecture.md]
+    P3[module.md]
+    P4[dependency-map.md]
+  end
+  Foundation --> P1
+  Foundation --> P2
+  Module --> P3
+  CrossCut --> P4
+```
+
+```mermaid
+sequenceDiagram
+  participant Planner
+  participant Assembler as Context Assembler
+  participant LLM
+  participant Linter
+
+  Planner->>Assembler: Page plan + source cards
+  Assembler->>Assembler: Select excerpts within token budget
+  Assembler->>LLM: Prompt + context
+  LLM-->>Assembler: Generated page content
+  Assembler->>Linter: Validate output
+  Linter-->>Assembler: Pass/fail + issues
+  alt lint passes
+    Assembler->>Planner: Emit final page
+  else lint fails
+    Assembler->>LLM: Retry with feedback
+  end
+```
+
+## Key Deliverables
+
+- LLM synthesis pipeline for each wiki page type (foundation, module, cross-cutting)
+- Incremental LLM enrichment that runs only for affected wiki pages selected by diff, bounded hierarchy propagation, and semantic propagation rules
+- Source card and code excerpt context assembly (token-budget aware)
+- Existing wiki page ingestion before regeneration
+- Page classification: generated, human-owned, mixed, unmanaged
+- Structured patch output format for wiki pages
+- Source citation enforcement (every material claim cites a path)
+- Contradiction and confidence metadata in generated pages
+- Human-maintained section preservation during regeneration
+- Stable page identity and ownership metadata in generated frontmatter
+- Merge strategy for back-fill and reconcile mode, not just fresh bootstrap generation
+- Prompt templates for each page archetype
+
+## Provider configuration contract
+
+The first production LLM boundary should be provider-agnostic and compatible with OpenAI-style chat completions so GitHub Actions and local runs can use OpenAI, compatible hosted providers, or local gateways without changing compiler code.
+
+Minimum `.llmwiki/config.json` shape:
+
+```jsonc
+{
+  "compiler": {
+    "mode": "deterministic", // deterministic | llm
+    "llm": {
+      "provider": "openai-compatible",
+      "base_url": "https://api.openai.com/v1",
+      "model": "gpt-4.1-mini",
+      "api_key_env": "LLMWIKI_LLM_API_KEY",
+      "system_prompt": "You compile source-grounded GitHub Wiki pages.",
+      "temperature": 0.1,
+      "max_output_tokens": 4000,
+      "timeout_ms": 60000,
+      "retries": 2
+    }
+  }
+}
+```
+
+Environment variables override config values for CI and secrets:
+
+| Environment variable | Purpose |
+|---|---|
+| `LLMWIKI_COMPILER_MODE` | Select `deterministic` or `llm` mode. |
+| `LLMWIKI_LLM_BASE_URL` | Override provider API base URL. |
+| `LLMWIKI_LLM_MODEL` | Override model name. |
+| `LLMWIKI_LLM_API_KEY` | Provider API key; never written to artifacts or logs. |
+| `LLMWIKI_LLM_SYSTEM_PROMPT` | Inline system prompt override. |
+| `LLMWIKI_LLM_SYSTEM_PROMPT_FILE` | Path to a system prompt file; useful for repo-maintained prompts. |
+| `LLMWIKI_LLM_TEMPERATURE` | Sampling temperature override. |
+| `LLMWIKI_LLM_MAX_OUTPUT_TOKENS` | Output token budget override. |
+
+Configuration precedence should be explicit and deterministic: CLI flags, when added, override environment variables; environment variables override `.llmwiki/config.json`; config overrides safe defaults. The API key must be read only from the configured environment variable and must never be persisted in scan artifacts, generated wiki pages, prompt-debug artifacts, or normal logs.
+
+GitHub Actions can enable LLM compilation by setting repository variables for non-secret values and a secret for the API key:
+
+```yaml
+env:
+  LLMWIKI_COMPILER_MODE: llm
+  LLMWIKI_LLM_BASE_URL: ${{ vars.LLMWIKI_LLM_BASE_URL }}
+  LLMWIKI_LLM_MODEL: ${{ vars.LLMWIKI_LLM_MODEL }}
+  LLMWIKI_LLM_API_KEY: ${{ secrets.LLMWIKI_LLM_API_KEY }}
+```
+
+Tests must use a deterministic mock provider and must not require network access or API keys.
+
+## Success Criteria
+
+- Generated wiki pages are useful without manual editing
+- Every factual claim cites source paths
+- Human-maintained sections survive regeneration unchanged
+- Existing mixed pages can be regenerated without losing preserved regions
+- Generated pages carry enough metadata to support future reconciliation and safe deletion
+- LLM output passes lint gates before acceptance
+- Untouched enriched pages remain byte-stable during incremental builds and do not incur model calls
+- Token budget stays within model context limits per page
+
+## Dependencies
+
+- Upstream: Production scanner (rich source cards), doc-validation (validated doc cards)
+- Downstream: Incremental mode (page patching), agent-integration (Agent-Context-Pack quality)
+
+## Open Questions
+
+- Which LLM provider(s) to support? (OpenAI, Anthropic, local models)
+- How much raw code should the compiler read per page?
+- Should compilation be parallelized across pages?
+- How to handle hallucination detection beyond lint gates?
+- Cost/latency budget for full bootstrap vs incremental compile?
+- What exact propagation thresholds should trigger re-enrichment of parent aggregate pages versus preserving existing content?
+- What is the minimum preservation contract for mixed human/generated pages?
+- Which page sections should be preserved structurally versus semantically merged?

package/docs/plans/production-scanner.md

@@ -0,0 +1,104 @@
+# Epic: Production Scanner
+
+## Summary
+
+Replace the current bootstrap scanner with a production-grade source analysis engine that performs AST-level extraction, detects framework-specific surfaces, builds import/dependency graphs, and maps tests to modules.
+
+## Architecture
+
+```mermaid
+flowchart TD
+  Source[Source Files] --> Parser[AST Parser]
+  Parser --> Symbols[Symbol Extraction]
+  Parser --> Imports[Import Resolution]
+  Parser --> Framework[Framework Detection]
+  Symbols --> Cards[Rich Source Cards]
+  Imports --> Graph[Import Graph]
+  Framework --> Routes[Route & API Surfaces]
+  Framework --> DB[Migration & ORM Models]
+  Graph --> AffectedPages[Affected-Page Graph]
+  Cards --> Output[Scanner Output]
+  Graph --> Output
+  Routes --> Output
+  DB --> Output
+  AffectedPages --> Output
+```
+
+```mermaid
+flowchart LR
+  subgraph Extractors
+    TS[TypeScript/JS]
+    Py[Python]
+    Go[Go]
+    Rust[Rust]
+  end
+  subgraph Frameworks
+    Express
+    Fastify
+    NestJS
+    NextJS[Next.js]
+    Hono
+    tRPC
+    GraphQL
+    OpenAPI
+  end
+  TS --> Frameworks
+  Py --> Frameworks
+```
+
+```mermaid
+graph TD
+  subgraph Import Graph
+    A[module-a.ts] --> B[module-b.ts]
+    A --> C[utils.ts]
+    B --> C
+    B --> D[db.ts]
+    D --> E[migrations/001.sql]
+  end
+  subgraph Test Mapping
+    T1[module-a.test.ts] -.-> A
+    T2[module-b.test.ts] -.-> B
+    T3[utils.test.ts] -.-> C
+  end
+```
+
+## Key Deliverables
+
+- TypeScript/JavaScript AST extraction (exports, classes, functions, types)
+- Framework detection: Express, Fastify, NestJS, Next.js, Hono, Koa, tRPC, GraphQL, OpenAPI
+- Route and API surface extraction
+- Database migration and ORM model detection
+- Import graph construction
+- Test-to-source mapping
+- Package script parsing
+- Affected-page graph for incremental mode
+
+## Success Criteria
+
+- Scanner produces rich source cards with symbol-level detail for supported languages
+- Framework-specific routes, middleware, and API surfaces are captured
+- Import graph enables transitive dependency reasoning
+- Test coverage mapping connects test files to the modules they exercise
+
+## Dependencies
+
+- Upstream: Current scaffold scanner (Milestone 1)
+- Downstream: Incremental mode (needs affected-page graph), LLM compiler (needs rich source cards)
+
+## Open Questions
+
+- Which AST parser(s) to use? (ts-morph, @swc/core, tree-sitter)
+- How to handle monorepo workspace boundaries?
+- Should Python/Go/Rust extractors be included in this epic or deferred?
+
+## Scope Decisions
+
+Decisions recorded during the foundation phase of this epic:
+
+1. **Regex extraction instead of AST parser**: The foundation phase uses regex-based extraction (`content.matchAll(…)`) rather than an AST parser (ts-morph / @swc/core / tree-sitter). This is a deliberate scope reduction for the initial phase; AST parsing is deferred to a follow-up.
+
+2. **DB/ORM detection (deterministic baseline)**: Scanner now performs deterministic path and regex-based detection for common migration files (`migrations/**`, SQL migration naming patterns, Prisma migrations) and model/entity declarations (Prisma `model`, TypeORM `@Entity`, Sequelize/Mongoose patterns). It records only safe metadata (paths, migration ids/names, model/entity names, hints) without copying SQL bodies or secret-like values.
+
+3. **Affected-page graph deferred**: `baseRef`/`headRef` options are scaffolded but ignored; the affected-page graph for incremental mode is deferred to the incremental-mode epic.
+
+4. **Framework detection partial**: Express, Fastify, Hono, and Next.js route-handler files are detected. NestJS, Koa, tRPC, GraphQL, and OpenAPI are deferred to a follow-up.

package/docs/plans/query-and-file-back.md

@@ -0,0 +1,103 @@
+# Epic: Query and File-Back Workflow
+
+## Summary
+
+Implement `repo-wiki query` and `repo-wiki search` as source-cited, wiki-first answer surfaces that treat the generated wiki as the first navigation layer before drilling into source cards and files for verification. Allow durable query answers to be filed back into the wiki as investigation or topic pages with full provenance, extending the Karpathy LLM Wiki pattern from compile-time knowledge capture to runtime knowledge compounding.
+
+## Architecture
+
+```mermaid
+sequenceDiagram
+  participant User
+  participant CLI
+  participant Search
+  participant Wiki
+  participant Source
+  participant Compiler
+
+  User->>CLI: repo-wiki query "How does auth work?"
+  CLI->>Search: Search Index.md, wiki pages, cards
+  Search-->>CLI: Candidate pages and source paths
+  CLI->>Wiki: Read relevant compiled pages
+  CLI->>Source: Verify material claims against source cards/files
+  CLI-->>User: Source-cited answer with confidence
+  alt answer is durable
+    User->>CLI: repo-wiki query --file-back "auth-investigation"
+    CLI->>Compiler: Create or update wiki page with provenance
+    Compiler->>Wiki: Write investigation page
+    CLI->>Wiki: Append query event to Log.md
+  end
+```
+
+```mermaid
+flowchart TD
+  Query["User query"] --> Index["Read Index.md"]
+  Index --> Pages["Rank candidate wiki pages"]
+  Pages --> WikiRead["Read compiled wiki pages"]
+  WikiRead --> Verify["Verify claims against source cards"]
+  Verify --> Answer["Source-cited answer + confidence"]
+  Answer --> FileBack{"File back?"}
+  FileBack -->|yes| Page["Investigation or topic page<br/>(provenance + query text + sources)"]
+  Page --> Log["Append to Log.md"]
+  FileBack -->|no| Done["Done"]
+```
+
+## Shipped command slice
+
+- `repo-wiki search <query>` — local ranked search over wiki pages.
+- `repo-wiki query <question>` — offline extractive answer assembly over ranked wiki pages plus graph provenance evidence.
+- `repo-wiki path <from> <to>` — deterministic shortest-path traversal over `.llmwiki/graph.json`.
+- `repo-wiki explain <node-or-page>` — focused local explanation tied to wiki page summaries and graph/source evidence.
+- All four commands support `--json` for machine-readable reuse.
+
+## Deferred file-back slice
+
+- `--file-back` flag on `query` to create or update a wiki page from a durable answer.
+- Filed-back pages include provenance (query text, answering commit, source paths, page state).
+- Query and file-back events appended to `Log.md` in the standard parseable format.
+- Optional hosted wording layered behind the same evidence path.
+- Mock/deterministic mode for tests (no hosted LLM required).
+- Query answers never treat stale or contradicted docs as authoritative.
+
+## Success Criteria
+
+- `repo-wiki search "query"` returns ranked wiki pages and evidence paths without external services.
+- `repo-wiki query`, `path`, and `explain` work without external services and expose JSON output.
+- Query and explain answers cite source paths for material claims when graph/wiki provenance is available.
+- Filed-back pages include provenance, query text, source paths, and page state in frontmatter.
+- The feature works in deterministic/mock mode for tests.
+- Query and file-back events appear in `Log.md` with the standard timestamp and operation type.
+
+## Acceptance Criteria (from PLAN.md)
+
+- Query answers cite source paths for material claims.
+- Filed-back pages include provenance, query text, source paths, and page state.
+- The feature works in deterministic/mock mode for tests.
+- `repo-wiki search "query"` returns ranked wiki pages and evidence paths.
+- Search can run without external services.
+- Optional provider integrations do not change core scan/compile behavior.
+
+## Filed-Back Page Frontmatter
+
+```yaml
+kind: investigation
+query: "How does auth work?"
+answered_at: "2026-05-10T14:30:00Z"
+source_commit: abc1234
+source_paths:
+  - src/auth.ts
+  - src/middleware/session.ts
+page_state: filed-back
+confidence: medium
+```
+
+## Dependencies
+
+- Upstream: wiki compiler, scanner, context assembler, LLM provider boundary.
+- Downstream: local search index (needed for scale), Log.md as parseable surface.
+
+## Open Questions
+
+- Should the first search backend be a built-in simple index, qmd integration, or MCP-first?
+- Should filed-back pages be proposed for review or written immediately?
+- How should confidence metadata propagate when a filed-back page is later updated by a new query?

package/docs/plans/search-index.md

@@ -0,0 +1,118 @@
+# Epic: Search Index
+
+## Summary
+
+Ship the first built-in local search path over compiled wiki pages so that `repo-wiki search` works fully offline today. Preserve enough metadata for later query/runtime routing, but defer external adapters until after the built-in page-first contract is stable.
+
+## Shipped built-in slice
+
+The shipped slice is intentionally bounded:
+
+- deterministic index artifact: `.llmwiki/search/index.json`
+- inputs: compiled wiki pages plus local metadata already present in page frontmatter and internal wiki links
+- ranking: deterministic page-first lexical scoring with evidence-oriented results
+- CLI: `repo-wiki search <query>` with text output for humans and `--json` for tools/agents
+- rebuild path: compile refreshes the search artifact, and `search` can rebuild it on demand from local wiki pages
+
+### Result contract
+
+Each result includes:
+
+- page title and page path
+- page `kind` / `page_state` when available
+- summary/snippet for quick routing
+- `source_paths` when available
+- lightweight graph context from inbound/outbound internal wiki links
+
+## Deferred after the shipped slice
+
+These remain explicitly deferred:
+
+- source-card/documentation-card indexing beyond what is already promoted into compiled wiki pages
+- section-level ranking as a primary retrieval contract
+- qmd, MCP, embedding, or hosted adapters
+- answer synthesis / `query` / file-back workflows
+
+## Architecture
+
+```mermaid
+flowchart TD
+  Wiki["Wiki pages (.llmwiki/wiki)"] --> Indexer["Index builder"]
+  Cards["Source + doc cards (.llmwiki/run)"] --> Indexer
+  Indexer --> Index["Local search index (.llmwiki/search/)"]
+  Index --> Simple["Built-in ranked text search"]
+  Index --> Adapter["Optional adapter layer"]
+  Adapter --> qmd["qmd backend (optional)"]
+  Adapter --> MCP["MCP endpoint (optional)"]
+  Simple --> Results["Ranked results<br/>(page, section, source paths)"]
+  qmd --> Results
+  MCP --> Results
+```
+
+```mermaid
+flowchart LR
+  subgraph Index entry
+    Title["page title"]
+    Category["page category"]
+    Summary["one-line summary"]
+    Body["searchable body text"]
+    Sources["source paths"]
+    Commit["source_commit"]
+  end
+```
+
+## Key Deliverables
+
+- Local index builder that runs after `compile` and indexes compiled wiki pages.
+- Built-in ranked text search with no external dependencies.
+- Index stored under `.llmwiki/search/` alongside run artifacts.
+- `repo-wiki search <query>` CLI command that returns ranked results with page title, category/kind, snippet, graph context, and source paths.
+- Deterministic full rebuilds that are compatible with later incremental indexing.
+- Index entries include page metadata: `kind`, `source_commit`, `source_paths`, `page_state`, and internal-link adjacency.
+
+## Success Criteria
+
+- `repo-wiki search "query"` returns ranked results without network access.
+- Search results include source paths so callers can drill into evidence.
+- The built-in shipped contract is stable enough for later query/runtime work to consume directly.
+- Index rebuild is fast enough to run after every compile in a typical repository.
+
+## Acceptance Criteria (from PLAN.md)
+
+- `repo-wiki search "query"` returns ranked wiki pages and evidence paths.
+- Search can run without external services.
+- Optional provider integrations do not change core scan/compile behavior.
+
+## Index Format
+
+```json
+{
+  "version": 1,
+  "wikiDir": "/repo/.llmwiki/wiki",
+  "sourceCommits": ["abc1234"],
+  "entries": [
+    {
+      "pagePath": "Architecture.md",
+      "title": "Architecture",
+      "kind": "foundation",
+      "pageState": "generated",
+      "summary": "High-level system design and data flow.",
+      "sourcePaths": ["src/compiler.ts", "src/scanner.ts"],
+      "outboundLinks": ["Module-scanner-ts.md"],
+      "inboundLinks": [],
+      "searchText": "architecture compiler scanner planner wiki"
+    }
+  ]
+}
+```
+
+## Dependencies
+
+- Upstream: wiki compiler (produces pages to index), scanner (provides source card metadata).
+- Downstream: query and file-back (uses search index for candidate routing).
+
+## Open Questions
+
+- When incremental mode becomes diff-minimal, what is the narrowest safe page-level reindex contract?
+- Should section-level ranking be added on top of the page-first contract, or remain a downstream concern for `query`?
+- If optional adapters arrive later, should qmd or MCP be the first external backend to implement?

package/docs/plans/trust-hardening.md

@@ -0,0 +1,74 @@
+# Epic: Trust Hardening
+
+## Summary
+
+Harden every surface where generated, scanned, or published content could leak secrets, reflect stale source truth, bypass severity policy, or let untrusted LLM patches reach the filesystem or a remote. This epic covers redaction, config schema validation, hash coverage, docs-lint blocking in the publish path, safe stale-page deletion, and publisher safety controls.
+
+## Architecture
+
+```mermaid
+flowchart TD
+  Scan["Scanner output"] --> Redact["Secret redaction"]
+  Redact --> Cards["Cards (no secrets)"]
+  Cards --> DocsLint["Docs linter"]
+  DocsLint --> Gate{"Error-level issues?"}
+  Gate -->|yes, configured to fail| Stop["Halt: do not compile/publish"]
+  Gate -->|no| Compiler["Compiler"]
+  Compiler --> PatchGate["Structured patch gate"]
+  PatchGate --> WikiPages["Wiki pages"]
+  WikiPages --> WikiLint["Wiki linter + health"]
+  WikiLint --> Publisher["Publisher"]
+  Publisher --> StaleDelete["Safe stale-page deletion<br/>(preserve human-owned + unmanaged)"]
+  StaleDelete --> Remote["Remote target"]
+```
+
+## Key Deliverables
+
+- **Docs-lint blocking**: make `repo-wiki run` fail before compile/publish when docs-lint reports error-level issues, controlled by config.
+- **Secret redaction**: redact known secret-like patterns from manifests, documentation cards, page contexts, log entries, and generated pages before writing.
+- **Config schema validation**: JSON schema for `.llmwiki/config.json` with clear validation errors on startup.
+- **Hash coverage**: every source card has a stable content hash or an explicit hash-failure reason recorded.
+- **Safe stale-page deletion**: publisher removes generated pages for deleted/renamed sources while preserving unmanaged and human-owned pages; ownership metadata from `page-ownership.ts` gates deletion.
+- **Severity policy**: lint severities for all gates are config-driven; no hardcoded unconditional failures except secret-like content.
+- **Sanitized remotes**: all remotes and URLs are validated and sanitized before display, logging, or writing.
+- **End-to-end fixture**: golden test covering `init → scan → plan → lint-docs → compile → lint → publish --dry-run`.
+
+## Success Criteria
+
+- Error-level docs-lint failures block compile and publish when configured.
+- No scan artifact or generated page contains known secret-like test patterns after redaction.
+- Publisher does not delete human-owned or unmanaged pages during stale cleanup.
+- `.llmwiki/config.json` schema validation surfaces clear errors on first use.
+- Every source card in the manifest has a `hash` field or a `hashError` field.
+
+## Acceptance Criteria (from PLAN.md)
+
+- Error-level docs lint failures can block run/publish according to config.
+- Scan output respects configured source filtering, including remaining `source.include` and nested-worktree edge cases.
+- Every source card has a stable hash or an explicit hash failure reason.
+- No scan artifact or generated page contains known secret-like patterns from fixtures.
+- Publisher removes stale generated pages without touching unmanaged or human-owned pages.
+- `npm test`, `npm run check`, `npm run coverage` all pass.
+- End-to-end fixture: `init → scan → plan → lint-docs → compile → lint → publish --dry-run`.
+
+## Severity Defaults
+
+| Gate | Default |
+|---|---|
+| Docs-lint error-level blocking | configurable (warn by default, fail when `fail_on_error: true`) |
+| Secret-like content | error (always blocks) |
+| Stale docs | warning |
+| Contradicted docs | error |
+| Broken relative links | warning |
+| Missing source commit | warning |
+
+## Dependencies
+
+- Upstream: docs linter, wiki linter, scanner, publisher.
+- Downstream: all epics — trust hardening is a prerequisite for safe LLM mode, incremental maintenance, and any publish to a public remote.
+
+## Open Questions
+
+- Which secret pattern list should be the canonical reference: `src/secret-patterns.ts` or an external policy file?
+- Should redaction replace matched text with `[REDACTED]` or remove the containing field entirely?
+- Should config schema validation block `repo-wiki run` or only `repo-wiki publish`?