llm-kb 0.4.0 → 0.4.2

package/PHASE4_SPEC.md

@@ -1,358 +0,0 @@
-# llm-kb — Phase 4: Farzapedia Pattern + Eval Loop
-
-> **The data flywheel is already spinning (v0.3.0):**
-> Query → Answer → Wiki updated → Next query answered from wiki → Faster, cheaper, compounding.
->
-> Phase 4 makes the flywheel bigger: proactive compilation + eval-driven refinement.
-
----
-
-## The Flywheel (what we already have)
-
-```
-         ┌─────────────┐
-         │  User asks   │
-         │  a question  │
-         └──────┬───────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  Agent answers from   │
-    │  wiki (fast) or       │
-    │  source files (slow)  │
-    └───────────┬───────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  wiki.md updated      │◄─── Haiku merges new knowledge
-    │  (topic-organized)    │     into existing wiki
-    └───────────┬───────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  Next similar query   │
-    │  answered from wiki   │──── 0 file reads, 2s instead of 25s
-    └───────────────────────┘
-```
-
-**Proven in production:**
-- First query about BNS 2023: 33s, 4 files read
-- Same question again: 2s, 0 files read, answered from wiki
-- Follow-up "tell me about mob lynching clause": instant, from wiki context
-
----
-
-## What Phase 4 adds
-
-### The problem with reactive-only wiki
-
-The wiki only knows what users have asked. If nobody asks about electronic evidence,
-that knowledge never makes it into the wiki. The first person to ask pays the full cost.
-
-### The Farzapedia insight
-
-> Compile the wiki **proactively** from all sources — BEFORE anyone asks.
-> Then every query is fast from day one.
-
-```
-Current (reactive only):
-  Sources exist → User asks → Agent reads sources → Answers → Wiki updated
-  Problem: first query for every topic is slow
-
-With compile (proactive + reactive):
-  Sources exist → Compile articles → User asks → Instant answer from articles
-  Plus: eval finds gaps → Articles refined → Even better answers
-```
-
----
-
-## Slices
-
-### Slice 1: Article compiler (part of `run`)
-
-**What:** After index is built, compile concept articles from all sources.
-Not a separate command — just step 3.5 in the `run` flow:
-
-```
-llm-kb run ./docs
-  1. Scan files
-  2. Parse PDFs
-  3. Build index          ← Haiku summarises sources
-  4. Compile articles      ← Sonnet synthesises concepts (NEW)
-  5. Start watchers
-  6. Start chat            ← Agent reads articles, not source files
-```
-
-**Skip logic (same as index):**
-- If `articles/` exists AND `articles/index.md` is newer than all source files → skip
-- If any source is newer OR articles/ missing → compile
-- First run always compiles. Subsequent runs are instant.
-
-**Input:**
-```
-.llm-kb/wiki/sources/
-  indian-penal-code-new.md (60 pages)
-  annotated-comparison-bns-ipc.md (21 pages)
-  evidence-act-new.md (40 pages)
-  ...
-```
-
-**Output:**
-```
-.llm-kb/wiki/articles/
-  index.md                          ← concept catalog with one-line descriptions
-  bns-2023-overview.md              ← what it is, structure, key changes
-  murder-and-homicide.md            ← Clauses 99-106, old vs new
-  mob-lynching.md                   ← Clause 101(2), new provision
-  electronic-evidence.md            ← Section 65B / BSB comparison
-  organised-crime.md                ← Clauses 109-110, new
-  sedition-removal.md               ← 124A removed, what replaces it
-  offences-against-women.md         ← Chapter V, new protections
-  ...
-```
-
-**Each article contains:**
-```markdown
-# Mob Lynching — BNS 2023, Clause 101(2)
-
-## Overview
-First-ever explicit criminalisation of mob lynching in Indian law...
-
-## The Provision
-When a group of 5+ persons acting in concert commits murder
-on discriminatory grounds (race, caste, community, sex, etc.)...
-
-## Punishment
-- Death, OR life imprisonment, OR minimum 7 years + fine
-- All members equally liable
-
-## Comparison with IPC
-IPC had no equivalent. Mob killings prosecuted under general S.302...
-
-## Related Articles
-- [[murder-and-homicide]] — general murder provisions
-- [[bns-2023-overview]] — the full new code
-- [[offences-against-women]] — other enhanced protections
-
-*Sources: indian penal code - new.md (p.137), Annotated comparison (p.15)*
-```
-
-**How it works:**
-1. Agent reads index.md to understand all sources
-2. Agent reads each source (or first ~2000 chars for large files)
-3. Agent identifies 10-30 key concepts across all sources
-4. Agent writes one article per concept with cross-references
-5. Agent writes articles/index.md catalog
-
-**Implementation:**
-- New function: `compileArticles(folder, sourcesDir, authStorage, modelId)`
-- Called from cli.ts `run` command, after buildIndex, before chat
-- Uses createAgentSession with read + write tools
-- AGENTS.md instructs the agent on article format, backlinks, source citations
-- Model: Sonnet (needs strong reasoning to synthesise across sources)
-
-**Definition of done:**
-- [ ] `run` compiles articles after index (with skip logic)
-- [ ] articles/index.md is a concept catalog with one-line descriptions
-- [ ] Each article has: overview, key details, source citations, related links
-- [ ] Articles are cross-referenced with [[article-name]] backlinks
-
----
-
-### Slice 2: Query uses articles
-
-**What:** When articles/ exists, the agent reads articles/index.md instead of source-index.
-It drills into specific articles rather than raw source files.
-
-**The navigation flow:**
-```
-Agent reads articles/index.md (concept catalog)
-  → Finds "mob-lynching.md" is relevant
-  → Reads articles/mob-lynching.md (small, focused, pre-synthesised)
-  → Answers instantly with cross-references
-  → NO raw source files read
-```
-
-**Implementation:**
-- Update buildQueryAgents() in query.ts
-- If articles/index.md exists: inject it into AGENTS.md, tell agent to use articles
-- Fallback: if no articles, use current source-index + wiki.md behaviour
-
-**Definition of done:**
-- [ ] Agent reads articles/index.md when available
-- [ ] Agent navigates to specific articles, not source files
-- [ ] Falls back to source-index when articles/ doesn't exist
-
----
-
-### Slice 3: Incremental article updates
-
-**What:** When a new file is dropped in, don't recompile everything.
-Update only the 2-3 articles affected by the new content.
-
-**Farza's quote:**
-> "The most magical thing now is as I add new things, the system updates
-> 2-3 different articles where it feels the context belongs, or just
-> creates a new article. Like a super genius librarian."
-
-**Flow:**
-```
-User drops "new-amendment-2024.pdf" into the folder
-  → Watcher: parse PDF → sources/new-amendment-2024.md
-  → Watcher: re-index (haiku)
-  → Watcher: read new source + articles/index.md
-  → Agent: "This affects mob-lynching.md and bns-2023-overview.md"
-  → Agent: updates those 2 articles + creates new-amendments-2024.md
-  → Agent: updates articles/index.md catalog
-```
-
-**Implementation:**
-- Update watcher.ts: after re-index, trigger incremental article update
-- Agent reads: new source file + articles/index.md
-- Agent decides: which articles to update, whether to create new ones
-- Uses Sonnet (needs reasoning about where new content fits)
-
-**Definition of done:**
-- [ ] New file → parse → re-index → update relevant articles
-- [ ] Agent updates 2-3 existing articles where content fits
-- [ ] Agent creates new article if topic is genuinely new
-- [ ] articles/index.md updated with any new entries
-
----
-
-### Slice 4: Eval — session analysis + article refinement
-
-**What:** Analyze session files to find quality issues and wiki gaps.
-Then fix the articles automatically.
-
-**Input:** `.llm-kb/sessions/*.jsonl` (raw conversation data)
-
-**What eval checks:**
-
-```
-CORRECTNESS
-  - Citation validity: does the source text support the claim?
-  - Consistency: does the answer contradict the sources?
-  
-PERFORMANCE
-  - Query time breakdown: wiki hit vs file reads
-  - Most-read source files (candidates for better articles)
-  - Wasted reads: files read but not cited
-  
-WIKI GAPS
-  - Questions that needed source files but should be in articles
-  - Articles that are incomplete (queries needed to read past them)
-  - Missing articles (topics asked about with no article)
-  
-INDEX ISSUES
-  - Wrong file reads: agent read irrelevant files (bad index summary)
-  - Redundant reads: same file read multiple times
-```
-
-**Output:** eval-report.md + automatic article patches
-
-```markdown
-# Eval Report — 2026-04-06
-
-## Summary
-15 sessions · 3 issues · 4 wiki gaps · estimated 120s saveable
-
-## 🔴 Correctness Issues
-1. Article "sedition-removal.md" says "retained" — source says "removed"
-   → AUTO-FIX: patched article
-
-## 🟡 Wiki Gaps (auto-filled)
-1. "Electronic evidence certification" — asked 4x, no article
-   → CREATED: articles/electronic-evidence-certification.md
-2. "CrPC comparison" — asked 3x, article was incomplete
-   → UPDATED: articles/crpc-comparison.md with missing sections
-
-## 🟢 Performance Insights
-- Wiki hit rate: 53% → 78% after gap fixes (estimated)
-- Most-read source: indian-penal-code-new.md (12 reads)
-  → Already well-covered by articles (reads are for exact quotes)
-- Wasted reads: 8 across 15 sessions (32% waste rate)
-```
-
-**Implementation:**
-- New command: `llm-kb eval`
-- Reads session JSONL files (full conversation data)
-- Code: extracts metrics (timing, file reads, citations)
-- LLM judge (Haiku): checks citation validity, identifies gaps
-- LLM writer (Haiku): patches articles with fixes
-- Writes eval-report.md
-
-**Definition of done:**
-- [ ] `llm-kb eval` reads sessions and writes eval-report.md
-- [ ] Flags: citation issues, consistency problems
-- [ ] Identifies: wiki gaps, performance bottlenecks
-- [ ] Auto-creates/patches articles for wiki gaps
-- [ ] Reports estimated time savings
-
----
-
-### Slice 5: The complete flywheel
-
-With all slices done, the full flywheel:
-
-```
-         ┌──────────────┐
-         │  COMPILE      │ Proactive: articles from all sources
-         │  (once/incr)  │
-         └──────┬────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  ARTICLES             │ Concept-organized, cross-referenced
-    │  articles/index.md    │ Agent navigates concepts, not files
-    │  articles/*.md        │
-    └───────────┬───────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  QUERY                │ User asks question
-    │  → reads article      │ Agent reads 1 small article, not 5 large sources
-    │  → instant answer     │ Sessions logged
-    └───────────┬───────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  EVAL                 │ Analyzes sessions
-    │  → finds gaps         │ Creates missing articles
-    │  → fixes errors       │ Patches wrong articles
-    │  → measures speed     │ Reports optimization opportunities
-    └───────────┬───────────┘
-                │
-                ▼
-    ┌───────────────────────┐
-    │  NEW FILE DROPPED     │ Watcher detects new source
-    │  → incremental update │ Updates 2-3 relevant articles
-    │  → index updated      │ New knowledge integrated
-    └───────────┬───────────┘
-                │
-                └──────────── back to QUERY (faster every cycle)
-```
-
-**The compounding effect:**
-- Day 1: compile articles from 9 PDFs → 15 articles
-- Day 2: 10 queries → eval finds 3 gaps → 3 articles added/fixed
-- Day 3: new PDF dropped → 2 articles updated
-- Day 4: 20 queries → 90% answered from articles (2s avg vs 25s)
-- Day 5: eval shows 95% wiki hit rate, 0 citation errors
-
----
-
-## Build Order
-
-| Slice | What | Effort | Priority |
-|---|---|---|---|
-| 1 | Article compiler (in `run`) | 2-3 hrs | 🔴 Do first |
-| 2 | Query reads articles | 30 min | 🔴 Immediate follow-up |
-| 3 | Incremental article updates (watcher) | 1-2 hrs | 🟡 This week |
-| 4 | `llm-kb eval` (session analysis + auto-fix) | 2-3 hrs | 🔴 The big one |
-| 5 | Full flywheel verification | Testing | 🟢 After all slices |
-
----
-
-*Phase 4 spec written April 6, 2026. DeltaXY.*
-*Inspired by Farzapedia (@FarzaTV) — Karpathy called it the best implementation of the LLM wiki pattern.*

package/SPEC.md

@@ -1,275 +0,0 @@
-# llm-kb — Product Spec
-
-> **One-liner:** Drop files into a folder. Get a knowledge base you can query.
-> **npm:** `npx llm-kb run ./my-documents`
-> **Status:** Phase 1 complete. Ingest pipeline + CLI.
-
----
-
-## Who Is This For
-
-A developer or technical researcher who has 20-200 documents (PDFs, spreadsheets, slide decks, notes) scattered across folders. They want to ask questions across all of them without building a RAG pipeline or setting up a vector database.
-
-**They will try this if:** it works in under 2 minutes with one command.
-**They will keep using it if:** the answers are good and the wiki compounds over time.
-**They will abandon it if:** setup is painful, it eats tokens without useful results, or it feels like a demo.
-
----
-
-## What Success Looks Like
-
-```bash
-npx llm-kb run ./research
-```
-
-Terminal output:
-```
-llm-kb v0.0.1
-
-Scanning ./research...
-  Found 12 files (12 PDF)
-  12 parsed
-
-  Building index...
-  Index built: .llm-kb/wiki/index.md
-
-  Output: ./research/.llm-kb/wiki/sources
-
-  Watching for new files... (Ctrl+C to stop)
-```
-
-Drop more files in while it's running. They get ingested automatically.
-
-**That's the whole first-run experience.** No config file. No API key prompt (uses Pi SDK auth). No Docker. Just point at a folder.
-
----
-
-## Commands
-
-### `llm-kb run <folder>` (Phase 1 ✅)
-
-The main command. Does everything:
-
-1. Scans the folder for PDF files
-2. Parses each PDF to markdown + JSON bounding boxes (via LiteParse)
-3. Skips already-parsed files (mtime check — re-runs are instant)
-4. Builds `index.md` from all parsed sources (via Pi SDK agent)
-5. Starts a file watcher on the folder (new PDFs get auto-ingested + re-indexed)
-
-**Data layout it creates inside the folder:**
-
-```
-./my-documents/
-├── (your original files — untouched)
-└── .llm-kb/
-    └── wiki/
-        ├── index.md
-        └── sources/
-            ├── report.md          ← spatial text layout
-            ├── report.json        ← per-word bounding boxes
-            └── ...
-```
-
-**Key decision:** `.llm-kb/` lives inside the user's folder, not in a global location. The knowledge base is co-located with the documents. Delete the folder, delete the KB. Copy the folder to another machine, the KB comes with it.
-
-### `llm-kb query <question>` (Phase 2)
-
-Query from the terminal without starting the web UI:
-
-```bash
-llm-kb query "what are the reserve requirements?" --folder ./research
-llm-kb query "compare Q3 vs Q4 guidance" --folder ./research --save
-```
-
-### `llm-kb status` (Phase 2)
-
-Show what's in the knowledge base.
-
-### `llm-kb eval` (Phase 4)
-
-Run the eval loop manually.
-
----
-
-## File Type Strategy
-
-**Key architectural decision: PDF is the only file type parsed at ingest time.** All other file types are handled dynamically by the Pi SDK agent at query time using pre-bundled libraries.
-
-### Why?
-
-- PDFs are binary, slow to parse, and need specialized libraries — worth pre-processing
-- Everything else (Excel, Word, PPT, CSV) — the Pi SDK agent can write a quick script to read them on demand
-- This eliminates 6 parser adapters, a router, and an adapter interface
-- The agent is smarter than a static adapter — it can decide what's relevant
-
-### PDF Parsing (Ingest Time)
-
-| Extension | Parser | Output | Bounding Boxes |
-|---|---|---|---|
-| `.pdf` | @llamaindex/liteparse | `.md` + `.json` | ✅ Yes |
-
-### Other File Types (Query Time — Agent Handles Dynamically)
-
-These libraries are pre-bundled in llm-kb and available to the agent via `NODE_PATH`:
-
-| Library | File Types |
-|---|---|
-| exceljs | `.xlsx`, `.xls` |
-| mammoth | `.docx` |
-| officeparser | `.pptx` |
-
-The agent's `AGENTS.md` context (injected via Pi SDK `agentsFilesOverride`) tells it which libraries are available and how to use them.
-
----
-
-## OCR Strategy
-
-Page-level routing — only scanned pages get OCR, native text pages are free and instant.
-
-```
-PDF Page → LiteParse classifies → native text? → keep local (free)
-                                → scanned?     → route to OCR
-```
-
-**OCR is off by default** (most PDFs have native text, avoids noisy Tesseract warnings).
-
-**Enable via env vars:**
-- `OCR_ENABLED=true` → local Tesseract.js (built into LiteParse)
-- `OCR_SERVER_URL=http://...` → remote Azure Document Intelligence bridge (faster, better quality)
-
-The OCR server is a separate project. llm-kb just calls it if the env var is set.
-
----
-
-## Auth & Model
-
-**No API key handling in llm-kb.** Uses Pi SDK's `createAgentSession()` with defaults:
-- Auth from `~/.pi/agent/auth.json` (existing Pi installation)
-- Model from Pi's settings (whatever the user has configured)
-- No config file, no env var for model selection
-
-```typescript
-const { session } = await createAgentSession({
-  cwd: folder,
-  resourceLoader: loader,   // injects AGENTS.md
-  tools: [readTool, bashTool, writeTool],
-  sessionManager: SessionManager.inMemory(),
-});
-```
-
----
-
-## Tech Stack (Phase 1 — What's Built)
-
-```
-TypeScript (strict)
-├── CLI: Commander
-├── Build: tsup (single bin/cli.js)
-├── Dev: Bun
-├── PDF parsing: @llamaindex/liteparse (local, bounding boxes)
-├── OCR: Tesseract.js (via LiteParse) or remote OCR server
-├── File watching: chokidar (debounced)
-├── Indexing: @mariozechner/pi-coding-agent (createAgentSession)
-├── Pre-bundled for agent: exceljs, mammoth, officeparser
-└── No database. No vector store. Files only.
-```
-
----
-
-## Project Structure (Actual)
-
-```
-llm-kb/
-├── bin/
-│   └── cli.js                 ← Built by tsup (single file)
-├── src/
-│   ├── cli.ts                 ← Commander entry point
-│   ├── scan.ts                ← Recursive folder scan + extension filter
-│   ├── pdf.ts                 ← LiteParse → .md + .json
-│   ├── indexer.ts             ← Pi SDK agent → writes index.md
-│   └── watcher.ts             ← chokidar file watcher (debounced)
-├── package.json
-├── tsconfig.json
-├── plan.md                    ← Emergent build plan
-├── README.md
-└── SPEC.md                    ← This file
-```
-
----
-
-## Constraints
-
-1. **Zero config for first run.** `npx llm-kb run ./folder` must work with Pi SDK auth. No config file needed. No init step.
-
-2. **No global state.** Everything lives in `.llm-kb/` inside the user's folder. Two different folders = two independent knowledge bases.
-
-3. **Original files are never modified.** Reads from the folder, writes only to `.llm-kb/`.
-
-4. **Graceful on bad files.** Corrupted PDF? Log a warning, skip it, continue. Show clean summary: `9 parsed, 1 failed`.
-
-5. **Token-conscious.** Pi SDK uses whatever model the user has configured. Indexing reads first ~500 chars of each file.
-
-6. **Offline-capable parsing.** PDF parsing runs locally via LiteParse. OCR is the only optional cloud dependency.
-
-7. **Works on Windows, Mac, Linux.** Tested on Windows. All Node.js, no shell scripts.
-
-8. **Skip up-to-date files.** Re-runs are instant — mtime check skips already-parsed PDFs.
-
----
-
-## What We're NOT Building (Yet)
-
-- **Multi-user auth.** Personal/team tool. No login.
-- **Cloud hosting.** Runs locally. Docker later.
-- **Real-time collaboration.** One user at a time.
-- **Vector search.** If the wiki outgrows context windows, we add it. Not before.
-- **Custom embeddings.** No ML pipeline. The LLM reads markdown.
-- **Config file.** Nothing reads it yet. Add when Phase 2/3 needs it (model selection, port, etc).
-- **Static file adapters.** No Excel/Word/PPT adapters. Pi SDK agent handles them dynamically.
-
----
-
-## Pre-Mortem: How This Fails
-
-| Failure | Why It Happened | Prevention |
-|---|---|---|
-| "Nobody tried it" | `npx` didn't work. Pi SDK not installed. | Clear prerequisites in README. |
-| "Tried it, too slow" | Indexing 20 PDFs took 5 minutes. | ✅ Progress bar. Skip up-to-date. Parse once. |
-| "Answers were bad" | Index summaries garbage → wrong files selected. | Test with real corpora. Eval loop (Phase 4). |
-| "Too expensive" | LLM burned tokens on indexing. | Agent reads first ~500 chars per file, not full content. |
-| "Broke on my files" | Encrypted PDF. 500MB file. | ✅ Graceful skip. Clean error messages. |
-| "Felt like a toy" | CLI only, no UI, no saved state. | Web UI in Phase 3. |
-
----
-
-## Build Order (Maps to Blog Series)
-
-| Phase | What | Status |
-|---|---|---|
-| **1** | CLI + PDF parsing + indexer + watcher | ✅ Done |
-| **2** | Query + Research sessions + terminal query command | Next |
-| **3** | Web UI (chat, upload, sources, activity) | Planned |
-| **4** | Eval (trace logger, eval session, report) | Planned |
-| **5** | Docker + deploy | Planned |
-| **6** | Citations (bounding boxes → highlight in PDF) | Planned |
-
----
-
-## Phase 1 — Definition of Done
-
-- [x] `llm-kb run ./folder` scans and parses PDFs
-- [x] Inline progress shows parsing status
-- [x] `.llm-kb/wiki/sources/` contains `.md` + `.json` per PDF
-- [x] `.llm-kb/wiki/index.md` generated with summary table
-- [x] File watcher auto-ingests new PDFs dropped into the folder
-- [x] Corrupt files skipped with warning, don't crash
-- [x] Re-runs skip up-to-date files (instant)
-- [x] OCR support via env var (local Tesseract or remote server)
-- [x] Auth via Pi SDK (no separate API key config)
-- [x] Works on Windows (tested), Mac/Linux (Node.js, should work)
-- [x] README has quickstart
-- [ ] Blog Part 2 written with real terminal output screenshots
-
----
-
-*Spec written April 4, 2026. Updated after Phase 1 build. DeltaXY.*