llm-kb 0.0.1 → 0.2.0

package/PHASE2_SPEC.md

@@ -0,0 +1,274 @@
+# llm-kb — Phase 2: Query Engine
+
+> **Goal:** `llm-kb query "question" --folder ./research` works from the terminal.
+> **Depends on:** Phase 1 (ingest pipeline — complete)
+> **Blog:** Part 3 of the series
+
+---
+
+## What Success Looks Like
+
+```bash
+llm-kb query "what are the reserve requirements?" --folder ./research
+```
+
+```
+Reading index... 12 sources
+Selected: reserve-policy.md, q3-results.md, board-deck.md
+Reading 3 files...
+
+Reserve requirements are defined in two documents:
+
+1. **Reserve Policy** (reserve-policy.md, p.3): Minimum reserve
+   ratio of 12% of total assets, reviewed quarterly.
+
+2. **Q3 Results** (q3-results.md, p.8): Current reserve ratio
+   is 14.2%, above the 12% minimum. Management notes this
+   provides a 2.2% buffer against regulatory changes.
+
+Sources: reserve-policy.md (p.3), q3-results.md (p.8)
+```
+
+That's the shape: file selection visible, citations inline, synthesis across sources.
+
+---
+
+## Two Modes
+
+### Query (read-only)
+
+```bash
+llm-kb query "what changed in Q4 guidance?" --folder ./research
+```
+
+The agent reads `index.md`, picks files, reads them, answers. **Cannot modify anything.** Tools: `createReadTool` only.
+
+### Research (read + write)
+
+```bash
+llm-kb query "compare pipeline coverage to revenue target" --folder ./research --save
+```
+
+Same as query, but the answer is also saved to `.llm-kb/wiki/outputs/`. The watcher detects the new file and re-indexes. Next query can reference the analysis.
+
+Tools: `createReadTool` + `createWriteTool` + `createBashTool`.
+
+The `--save` flag switches from query mode to research mode.
+
+---
+
+## Architecture
+
+Same pattern as the indexer — a Pi SDK session with different tools:
+
+```typescript
+export async function query(
+  folder: string,
+  question: string,
+  options: { save?: boolean }
+) {
+  const sourcesDir = join(folder, ".llm-kb", "wiki", "sources");
+  const outputsDir = join(folder, ".llm-kb", "wiki", "outputs");
+
+  // Build AGENTS.md for query context
+  const agentsContent = buildQueryAgents(sourcesDir, options.save);
+
+  const loader = new DefaultResourceLoader({
+    cwd: folder,
+    agentsFilesOverride: (current) => ({
+      agentsFiles: [
+        ...current.agentsFiles,
+        { path: ".llm-kb/AGENTS.md", content: agentsContent },
+      ],
+    }),
+  });
+  await loader.reload();
+
+  const tools = [createReadTool(folder)];
+  if (options.save) {
+    tools.push(createWriteTool(folder), createBashTool(folder));
+  }
+
+  const { session } = await createAgentSession({
+    cwd: folder,
+    resourceLoader: loader,
+    tools,
+    sessionManager: SessionManager.inMemory(),
+    settingsManager: SettingsManager.inMemory({
+      compaction: { enabled: false },
+    }),
+  });
+
+  // Stream output to terminal
+  session.subscribe((event) => {
+    if (
+      event.type === "message_update" &&
+      event.assistantMessageEvent.type === "text_delta"
+    ) {
+      process.stdout.write(event.assistantMessageEvent.delta);
+    }
+  });
+
+  await session.prompt(question);
+  session.dispose();
+}
+```
+
+### The Query AGENTS.md
+
+The injected `AGENTS.md` for query mode tells the agent:
+
+```markdown
+# llm-kb Knowledge Base — Query Mode
+
+## How to answer questions
+
+1. FIRST read .llm-kb/wiki/index.md to see all available sources
+2. Based on the question, select the most relevant source files
+3. Read those files in full (not just the first 500 chars)
+4. Answer with inline citations: (filename, page/section)
+5. If the answer requires cross-referencing, read additional files
+6. Prefer primary sources over previous analyses in outputs/
+
+## Available sources
+(dynamically generated list of .md files in sources/)
+
+## Available libraries for non-PDF files
+- exceljs — for .xlsx/.xls
+- mammoth — for .docx
+- officeparser — for .pptx
+Write a quick Node.js script via bash to read these when needed.
+
+## Rules
+- Always cite sources with filename and page number
+- If you can't find the answer, say so — don't hallucinate
+- Read the FULL file, not just the beginning
+```
+
+For research mode, add:
+
+```markdown
+## Research Mode
+You can save your analysis to .llm-kb/wiki/outputs/.
+Use a descriptive filename (e.g., coverage-analysis.md).
+The file watcher will detect it and update the index.
+```
+
+---
+
+## CLI Integration
+
+Add `query` command to Commander:
+
+```typescript
+program
+  .command("query")
+  .description("Ask a question across your knowledge base")
+  .argument("<question>", "Your question")
+  .option("--folder <path>", "Path to document folder", ".")
+  .option("--save", "Save the answer to wiki/outputs/ (research mode)")
+  .action(async (question, options) => {
+    const folder = resolve(options.folder);
+
+    // Check if .llm-kb exists
+    if (!existsSync(join(folder, ".llm-kb"))) {
+      console.error(chalk.red("No knowledge base found. Run 'llm-kb run' first."));
+      process.exit(1);
+    }
+
+    await query(folder, question, { save: options.save });
+  });
+```
+
+---
+
+## Trace Logging (Prep for Eval — Phase 4)
+
+Every query gets logged to `.llm-kb/traces/`:
+
+```json
+{
+  "timestamp": "2026-04-05T14:30:00Z",
+  "question": "what are the reserve requirements?",
+  "mode": "query",
+  "filesRead": ["index.md", "reserve-policy.md", "q3-results.md"],
+  "filesAvailable": ["reserve-policy.md", "q3-results.md", "board-deck.md", "pipeline.md"],
+  "answer": "Reserve requirements are defined in two documents...",
+  "citations": [
+    { "file": "reserve-policy.md", "location": "p.3", "claim": "Minimum reserve ratio of 12%" },
+    { "file": "q3-results.md", "location": "p.8", "claim": "Current reserve ratio is 14.2%" }
+  ],
+  "tokensUsed": 3800,
+  "durationMs": 4200,
+  "model": "claude-sonnet-4"
+}
+```
+
+Implementation: wrap the session to intercept tool calls and capture which files were read. Save trace JSON after session completes.
+
+The eval agent (Phase 4) reads these traces to check citations against sources.
+
+---
+
+## Streaming Output
+
+Terminal query should stream — the user sees the answer appear word by word, not wait for the full response. The `session.subscribe()` handler writes deltas to stdout.
+
+For the `run` command (when we add query to the web UI in Phase 3), streaming goes through the Vercel AI SDK protocol.
+
+---
+
+## Constraints
+
+1. **Query must work without the web server running.** `llm-kb query` is standalone — it reads `.llm-kb/` directly. No dependency on `llm-kb run`.
+
+2. **Read-only by default.** Query mode cannot modify files. Only `--save` enables write.
+
+3. **Index must exist.** If `.llm-kb/wiki/index.md` doesn't exist, error out: "No knowledge base found. Run 'llm-kb run' first."
+
+4. **Graceful on empty results.** If the agent can't find relevant files, it should say "I couldn't find sources relevant to this question" — not hallucinate.
+
+5. **Token-conscious.** The agent reads index.md (~200 tokens for 50 sources) first, then only the files it selects (3-7 typically). Don't read all sources.
+
+---
+
+## Build Order (Slices)
+
+| Slice | What | Demoable? |
+|---|---|---|
+| 1 | `query` command + read-only session + streaming | ✅ Ask questions, get answers |
+| 2 | `--save` flag + research mode + write to outputs/ | ✅ Answers compound in wiki |
+| 3 | Trace logging (JSON per query) | Prep for eval |
+| 4 | `status` command (show KB stats) | ✅ Nice-to-have |
+
+---
+
+## Definition of Done
+
+- [ ] `llm-kb query "question" --folder ./research` returns a cited answer
+- [ ] Answer streams to terminal (word by word, not all at once)
+- [ ] Agent reads index.md first, then selects and reads relevant source files
+- [ ] `--save` flag saves the answer to `.llm-kb/wiki/outputs/`
+- [ ] Saved answers get detected by watcher and re-indexed
+- [ ] Query traces logged to `.llm-kb/traces/` as JSON
+- [ ] Error if no `.llm-kb/` exists ("run 'llm-kb run' first")
+- [ ] Non-PDF files (Excel, Word) readable by agent via bundled libraries
+- [ ] Blog Part 3 written with real terminal output
+
+---
+
+## What This Enables
+
+With query working, the demo becomes:
+
+```bash
+npx llm-kb run ./my-documents    # ingest
+llm-kb query "what changed?"     # ask
+llm-kb query "compare X vs Y" --save  # research (compounds)
+```
+
+Three commands. Ingest → Query → Research. That's a product, not a script.
+
+---
+
+*Phase 2 spec written April 4, 2026. DeltaXY.*

package/README.md

@@ -1,21 +1,133 @@
 # llm-kb
 
-LLM-powered knowledge base. Drop documents, build a wiki, ask questions.
+Drop files into a folder. Get a knowledge base you can query.
 
 Inspired by [Karpathy's LLM Knowledge Bases](https://x.com/karpathy/status/2039805659525644595).
 
+## Quick Start
+
 ```bash
-npx llm-kb run ./my-documents
+npm install -g llm-kb
+llm-kb run ./my-documents
 ```
 
+That's it. Your PDFs get parsed to markdown, an index is built, and a file watcher keeps it up to date.
+
+### Prerequisites
+
+- **Node.js 18+**
+- **Pi SDK** installed and authenticated (`npm install -g @mariozechner/pi-coding-agent` + run `pi` once to set up auth)
+
+Pi handles the LLM auth — no separate API key configuration needed.
+
 ## What It Does
 
-- **Ingest** — drop PDFs, Excel, Word, PowerPoint, images, or text into a folder
-- **Parse** — automatically converts to markdown (LiteParse for PDFs, ExcelJS for spreadsheets, Mammoth for Word)
-- **Index** — LLM reads all sources, maintains an index with summaries and topics
-- **Query** — ask questions, get answers with citations
-- **Research** — answers saved back to the wiki, compounding knowledge
-- **Eval** — checks answers against sources, reports failures
+### Ingest
+
+```bash
+llm-kb run ./my-documents
+```
+
+```
+llm-kb v0.2.0
+
+Scanning ./my-documents...
+  Found 9 files (9 PDF)
+  9 parsed
+
+  Building index...
+  Index built: .llm-kb/wiki/index.md
+
+  Output: ./my-documents/.llm-kb/wiki/sources
+
+  Watching for new files... (Ctrl+C to stop)
+```
+
+1. **Scans** the folder for PDFs
+2. **Parses** each PDF to markdown + bounding boxes (using [LiteParse](https://github.com/run-llama/liteparse))
+3. **Builds an index** — Pi SDK agent reads all sources and writes `index.md` with summaries
+4. **Watches** — drop a new PDF in while it's running, it gets parsed and indexed automatically
+
+### Query
+
+```bash
+# From inside the documents folder (auto-detects .llm-kb/)
+llm-kb query "what are the key findings?"
+
+# From anywhere, with explicit folder
+llm-kb query "compare Q3 vs Q4" --folder ./my-documents
+
+# Research mode — saves the answer to wiki/outputs/ and re-indexes
+llm-kb query "summarize all revenue data" --save
+```
+
+The agent reads `index.md`, selects relevant files, and streams a cited answer to the terminal.
+
+**Query mode** — read-only. The agent can only read your files.
+**Research mode** (`--save`) — read + write + bash. The agent saves answers to `outputs/`, re-indexes, and can write scripts to read Excel/Word files. Answers compound over time.
+
+### What It Creates
+
+```
+./my-documents/
+├── (your files — untouched)
+└── .llm-kb/
+    └── wiki/
+        ├── index.md              ← summary of all sources
+        └── sources/
+            ├── report.md         ← parsed text (spatial layout)
+            ├── report.json       ← bounding boxes (for citations)
+            └── ...
+```
+
+Your original files are never modified. Delete `.llm-kb/` to start fresh.
+
+## OCR for Scanned PDFs
+
+Most PDFs have native text — they just work. For scanned PDFs:
+
+**Local (default when enabled):**
+```bash
+OCR_ENABLED=true llm-kb run ./my-documents
+```
+Uses Tesseract.js (built-in, slower but works everywhere).
+
+**Remote OCR server (faster, better quality):**
+```bash
+OCR_SERVER_URL="http://localhost:8080/ocr?key=YOUR_KEY" llm-kb run ./my-documents
+```
+Routes scanned pages to an Azure Document Intelligence bridge. Native-text pages still processed locally (free).
+
+## Non-PDF Files
+
+PDFs are parsed at ingest time. Other file types (Excel, Word, PowerPoint, CSV, images) are handled dynamically by the Pi SDK agent at query time — it writes quick scripts using pre-bundled libraries:
+
+| Library | File Types |
+|---|---|
+| exceljs | `.xlsx`, `.xls` |
+| mammoth | `.docx` |
+| officeparser | `.pptx` |
+
+No separate install needed — all bundled with llm-kb.
+
+## How It Works
+
+- **PDF parsing** — `@llamaindex/liteparse` extracts text with spatial layout + per-word bounding boxes. Runs locally, no cloud calls.
+- **Indexing** — Pi SDK `createAgentSession` reads each source and generates a summary table in `index.md`.
+- **File watching** — `chokidar` watches the folder. New/changed PDFs trigger re-parse + re-index (debounced for batch drops).
+- **Auth** — uses Pi SDK's auth storage (`~/.pi/agent/auth.json`). No API keys in your project.
+
+## Development
+
+```bash
+git clone https://github.com/satish860/llm-kb
+cd llm-kb
+bun install
+bun run build
+npm link
+
+llm-kb run ./test-folder
+```
 
 ## Tutorial
 

package/SPEC.md

@@ -0,0 +1,275 @@
+# llm-kb — Product Spec
+
+> **One-liner:** Drop files into a folder. Get a knowledge base you can query.
+> **npm:** `npx llm-kb run ./my-documents`
+> **Status:** Phase 1 complete. Ingest pipeline + CLI.
+
+---
+
+## Who Is This For
+
+A developer or technical researcher who has 20-200 documents (PDFs, spreadsheets, slide decks, notes) scattered across folders. They want to ask questions across all of them without building a RAG pipeline or setting up a vector database.
+
+**They will try this if:** it works in under 2 minutes with one command.
+**They will keep using it if:** the answers are good and the wiki compounds over time.
+**They will abandon it if:** setup is painful, it eats tokens without useful results, or it feels like a demo.
+
+---
+
+## What Success Looks Like
+
+```bash
+npx llm-kb run ./research
+```
+
+Terminal output:
+```
+llm-kb v0.0.1
+
+Scanning ./research...
+  Found 12 files (12 PDF)
+  12 parsed
+
+  Building index...
+  Index built: .llm-kb/wiki/index.md
+
+  Output: ./research/.llm-kb/wiki/sources
+
+  Watching for new files... (Ctrl+C to stop)
+```
+
+Drop more files in while it's running. They get ingested automatically.
+
+**That's the whole first-run experience.** No config file. No API key prompt (uses Pi SDK auth). No Docker. Just point at a folder.
+
+---
+
+## Commands
+
+### `llm-kb run <folder>` (Phase 1 ✅)
+
+The main command. Does everything:
+
+1. Scans the folder for PDF files
+2. Parses each PDF to markdown + JSON bounding boxes (via LiteParse)
+3. Skips already-parsed files (mtime check — re-runs are instant)
+4. Builds `index.md` from all parsed sources (via Pi SDK agent)
+5. Starts a file watcher on the folder (new PDFs get auto-ingested + re-indexed)
+
+**Data layout it creates inside the folder:**
+
+```
+./my-documents/
+├── (your original files — untouched)
+└── .llm-kb/
+    └── wiki/
+        ├── index.md
+        └── sources/
+            ├── report.md          ← spatial text layout
+            ├── report.json        ← per-word bounding boxes
+            └── ...
+```
+
+**Key decision:** `.llm-kb/` lives inside the user's folder, not in a global location. The knowledge base is co-located with the documents. Delete the folder, delete the KB. Copy the folder to another machine, the KB comes with it.
+
+### `llm-kb query <question>` (Phase 2)
+
+Query from the terminal without starting the web UI:
+
+```bash
+llm-kb query "what are the reserve requirements?" --folder ./research
+llm-kb query "compare Q3 vs Q4 guidance" --folder ./research --save
+```
+
+### `llm-kb status` (Phase 2)
+
+Show what's in the knowledge base.
+
+### `llm-kb eval` (Phase 4)
+
+Run the eval loop manually.
+
+---
+
+## File Type Strategy
+
+**Key architectural decision: PDF is the only file type parsed at ingest time.** All other file types are handled dynamically by the Pi SDK agent at query time using pre-bundled libraries.
+
+### Why?
+
+- PDFs are binary, slow to parse, and need specialized libraries — worth pre-processing
+- Everything else (Excel, Word, PPT, CSV) — the Pi SDK agent can write a quick script to read them on demand
+- This eliminates 6 parser adapters, a router, and an adapter interface
+- The agent is smarter than a static adapter — it can decide what's relevant
+
+### PDF Parsing (Ingest Time)
+
+| Extension | Parser | Output | Bounding Boxes |
+|---|---|---|---|
+| `.pdf` | @llamaindex/liteparse | `.md` + `.json` | ✅ Yes |
+
+### Other File Types (Query Time — Agent Handles Dynamically)
+
+These libraries are pre-bundled in llm-kb and available to the agent via `NODE_PATH`:
+
+| Library | File Types |
+|---|---|
+| exceljs | `.xlsx`, `.xls` |
+| mammoth | `.docx` |
+| officeparser | `.pptx` |
+
+The agent's `AGENTS.md` context (injected via Pi SDK `agentsFilesOverride`) tells it which libraries are available and how to use them.
+
+---
+
+## OCR Strategy
+
+Page-level routing — only scanned pages get OCR, native text pages are free and instant.
+
+```
+PDF Page → LiteParse classifies → native text? → keep local (free)
+                                → scanned?     → route to OCR
+```
+
+**OCR is off by default** (most PDFs have native text, avoids noisy Tesseract warnings).
+
+**Enable via env vars:**
+- `OCR_ENABLED=true` → local Tesseract.js (built into LiteParse)
+- `OCR_SERVER_URL=http://...` → remote Azure Document Intelligence bridge (faster, better quality)
+
+The OCR server is a separate project. llm-kb just calls it if the env var is set.
+
+---
+
+## Auth & Model
+
+**No API key handling in llm-kb.** Uses Pi SDK's `createAgentSession()` with defaults:
+- Auth from `~/.pi/agent/auth.json` (existing Pi installation)
+- Model from Pi's settings (whatever the user has configured)
+- No config file, no env var for model selection
+
+```typescript
+const { session } = await createAgentSession({
+  cwd: folder,
+  resourceLoader: loader,   // injects AGENTS.md
+  tools: [readTool, bashTool, writeTool],
+  sessionManager: SessionManager.inMemory(),
+});
+```
+
+---
+
+## Tech Stack (Phase 1 — What's Built)
+
+```
+TypeScript (strict)
+├── CLI: Commander
+├── Build: tsup (single bin/cli.js)
+├── Dev: Bun
+├── PDF parsing: @llamaindex/liteparse (local, bounding boxes)
+├── OCR: Tesseract.js (via LiteParse) or remote OCR server
+├── File watching: chokidar (debounced)
+├── Indexing: @mariozechner/pi-coding-agent (createAgentSession)
+├── Pre-bundled for agent: exceljs, mammoth, officeparser
+└── No database. No vector store. Files only.
+```
+
+---
+
+## Project Structure (Actual)
+
+```
+llm-kb/
+├── bin/
+│   └── cli.js                 ← Built by tsup (single file)
+├── src/
+│   ├── cli.ts                 ← Commander entry point
+│   ├── scan.ts                ← Recursive folder scan + extension filter
+│   ├── pdf.ts                 ← LiteParse → .md + .json
+│   ├── indexer.ts             ← Pi SDK agent → writes index.md
+│   └── watcher.ts             ← chokidar file watcher (debounced)
+├── package.json
+├── tsconfig.json
+├── plan.md                    ← Emergent build plan
+├── README.md
+└── SPEC.md                    ← This file
+```
+
+---
+
+## Constraints
+
+1. **Zero config for first run.** `npx llm-kb run ./folder` must work with Pi SDK auth. No config file needed. No init step.
+
+2. **No global state.** Everything lives in `.llm-kb/` inside the user's folder. Two different folders = two independent knowledge bases.
+
+3. **Original files are never modified.** Reads from the folder, writes only to `.llm-kb/`.
+
+4. **Graceful on bad files.** Corrupted PDF? Log a warning, skip it, continue. Show clean summary: `9 parsed, 1 failed`.
+
+5. **Token-conscious.** Pi SDK uses whatever model the user has configured. Indexing reads first ~500 chars of each file.
+
+6. **Offline-capable parsing.** PDF parsing runs locally via LiteParse. OCR is the only optional cloud dependency.
+
+7. **Works on Windows, Mac, Linux.** Tested on Windows. All Node.js, no shell scripts.
+
+8. **Skip up-to-date files.** Re-runs are instant — mtime check skips already-parsed PDFs.
+
+---
+
+## What We're NOT Building (Yet)
+
+- **Multi-user auth.** Personal/team tool. No login.
+- **Cloud hosting.** Runs locally. Docker later.
+- **Real-time collaboration.** One user at a time.
+- **Vector search.** If the wiki outgrows context windows, we add it. Not before.
+- **Custom embeddings.** No ML pipeline. The LLM reads markdown.
+- **Config file.** Nothing reads it yet. Add when Phase 2/3 needs it (model selection, port, etc).
+- **Static file adapters.** No Excel/Word/PPT adapters. Pi SDK agent handles them dynamically.
+
+---
+
+## Pre-Mortem: How This Fails
+
+| Failure | Why It Happened | Prevention |
+|---|---|---|
+| "Nobody tried it" | `npx` didn't work. Pi SDK not installed. | Clear prerequisites in README. |
+| "Tried it, too slow" | Indexing 20 PDFs took 5 minutes. | ✅ Progress bar. Skip up-to-date. Parse once. |
+| "Answers were bad" | Index summaries garbage → wrong files selected. | Test with real corpora. Eval loop (Phase 4). |
+| "Too expensive" | LLM burned tokens on indexing. | Agent reads first ~500 chars per file, not full content. |
+| "Broke on my files" | Encrypted PDF. 500MB file. | ✅ Graceful skip. Clean error messages. |
+| "Felt like a toy" | CLI only, no UI, no saved state. | Web UI in Phase 3. |
+
+---
+
+## Build Order (Maps to Blog Series)
+
+| Phase | What | Status |
+|---|---|---|
+| **1** | CLI + PDF parsing + indexer + watcher | ✅ Done |
+| **2** | Query + Research sessions + terminal query command | Next |
+| **3** | Web UI (chat, upload, sources, activity) | Planned |
+| **4** | Eval (trace logger, eval session, report) | Planned |
+| **5** | Docker + deploy | Planned |
+| **6** | Citations (bounding boxes → highlight in PDF) | Planned |
+
+---
+
+## Phase 1 — Definition of Done
+
+- [x] `llm-kb run ./folder` scans and parses PDFs
+- [x] Inline progress shows parsing status
+- [x] `.llm-kb/wiki/sources/` contains `.md` + `.json` per PDF
+- [x] `.llm-kb/wiki/index.md` generated with summary table
+- [x] File watcher auto-ingests new PDFs dropped into the folder
+- [x] Corrupt files skipped with warning, don't crash
+- [x] Re-runs skip up-to-date files (instant)
+- [x] OCR support via env var (local Tesseract or remote server)
+- [x] Auth via Pi SDK (no separate API key config)
+- [x] Works on Windows (tested), Mac/Linux (Node.js, should work)
+- [x] README has quickstart
+- [ ] Blog Part 2 written with real terminal output screenshots
+
+---
+
+*Spec written April 4, 2026. Updated after Phase 1 build. DeltaXY.*