@zosmaai/pi-llm-wiki 0.1.6 → 0.2.0

package/skills/llm-wiki/SKILL.md

@@ -1,426 +1,152 @@
 ---
 name: llm-wiki
-description: Build and maintain a persistent, interlinked Obsidian-compatible markdown wiki using Karpathy's LLM Wiki pattern. Use for ingest, query, lint, discover, and status operations on personal or company knowledge bases.
+description: Build and maintain a persistent, interlinked Obsidian-compatible markdown wiki using Karpathy's LLM Wiki pattern. Extension-backed with auto-generated metadata, guardrails, and 10 custom tools.
 ---
 
 # LLM Wiki for Pi
 
-You are a disciplined wiki maintainer. You manage a persistent knowledge base following Andrej Karpathy's LLM Wiki pattern: raw sources go in `raw/`, the wiki (interlinked markdown) lives in `wiki/`, and the schema is defined here.
+You are a disciplined wiki maintainer. The extension handles all mechanical work — you focus on synthesis, reasoning, and knowledge organization.
 
-The human curates sources and asks questions. You do everything else — reading, summarizing, cross-referencing, synthesizing, linting, and maintaining.
-
----
-
-## Architecture
+## Architecture (4 Layers)
 
 ```
 WIKI_ROOT/
-├── raw/                   # Immutable source documents (human-owned)
-│   ├── articles/          # Web articles, blog posts (markdown)
-│   ├── papers/            # Research papers (PDF)
-│   ├── notes/             # Personal notes, meeting transcripts
-│   ├── slack/             # Slack / Discord thread exports
-│   ├── email/             # Important email threads
-│   └── assets/            # Downloaded images (Obsidian attachment folder)
-├── wiki/                  # LLM-written & maintained (you own this)
-│   ├── INDEX.md           # Master catalog of all pages
-│   ├── LOG.md             # Append-only chronological activity log
-│   ├── DASHBOARD.md       # Obsidian Dataview dashboard
-│   ├── entities/          # People, organizations, tools, projects
-│   ├── concepts/          # Ideas, patterns, frameworks, methodologies
-│   ├── sources/           # One summary per ingested source document
-│   ├── syntheses/         # Cross-cutting analyses, comparisons, insights
-│   └── changes/           # Change detection records (company wiki)
-├── outputs/               # Reports, battlecards, digests, lint results
-├── config.yaml            # Wiki configuration (topics, feeds, settings)
-└── .discoveries/          # Auto-discovery metadata
-    ├── history.json       # Sources already processed (dedup)
-    └── gaps.json          # Knowledge gaps discovered by lint
+├── raw/sources/SRC-*/       # Immutable source packets (extension-owned)
+│   ├── manifest.json
+│   ├── original/
+│   ├── extracted.md
+│   └── attachments/
+├── wiki/                     # Editable knowledge pages (you own this)
+│   ├── sources/              # One summary per source
+│   ├── entities/             # People, orgs, tools, products
+│   ├── concepts/             # Ideas, patterns, frameworks
+│   ├── syntheses/            # Cross-cutting analyses
+│   └── analyses/             # Durable query answers
+├── meta/                     # Auto-generated (extension-owned)
+│   ├── registry.json         # Master page catalog
+│   ├── backlinks.json        # Inbound link map
+│   ├── index.md              # Human-readable catalog
+│   ├── log.md                # Activity log
+│   └── events.jsonl          # Structured event stream
+└── .wiki/                    # Config and templates
+    ├── config.json
+    └── templates/
 ```
 
-### Three Layers
-
-| Layer           | Path                          | Owner      | Purpose                                               |
-| --------------- | ----------------------------- | ---------- | ----------------------------------------------------- |
-| **Raw Sources** | `raw/`                        | Human      | Immutable source of truth. You READ ONLY.             |
-| **The Wiki**    | `wiki/`                       | You (LLM)  | Interlinked markdown pages. You write and maintain.   |
-| **The Schema**  | This SKILL.md + `config.yaml` | Co-evolved | Rules, conventions, workflows. Updated as needs grow. |
-
----
-
 ## Golden Rules
 
-1. **RAW IS IMMUTABLE.** Never edit, delete, or rename files in `raw/`. Every wiki claim must trace back to a raw source.
-2. **YOU OWN THE WIKI.** Never ask the human to edit wiki pages. You create, update, and cross-reference everything.
-3. **ONE FILE PER THING.** Each entity, concept, source gets its own `.md` file. Keep pages under ~500 lines.
-4. **CROSS-REFERENCE EVERYTHING.** Every page needs at least 2 `[[wikilinks]]` to other pages. No orphans.
-5. **KEEP INDEX CURRENT.** Every page create/update/delete → update `INDEX.md` immediately.
-6. **LOG ALL ACTIVITY.** Every action → append to `LOG.md` with timestamp.
-7. **CITE SOURCES.** Every claim links back to its raw source file. Never fabricate.
-8. **FLAG CONTRADICTIONS.** New info contradicting existing content → document both, note the conflict.
-
----
+1. **RAW IS IMMUTABLE.** Never edit `raw/`. Use `wiki_capture_source` to add sources.
+2. **META IS AUTO-GENERATED.** Never edit `meta/`. The extension rebuilds it automatically.
+3. **YOU OWN THE WIKI.** Create, update, and cross-reference everything in `wiki/`.
+4. **ONE FILE PER THING.** Each entity, concept, source gets its own `.md` file.
+5. **CROSS-REFERENCE EVERYTHING.** Every page needs at least 2 `[[wikilinks]]`.
+6. **CITE SOURCES.** Every claim links back to its raw source packet.
+7. **FLAG CONTRADICTIONS.** When sources disagree, document both sides.
+
+## How the Extension Helps You
+
+| Task               | Before (skill-only)          | Now (extension-backed)                |
+| ------------------ | ---------------------------- | ------------------------------------- |
+| Track ingestion    | Manual `history.json`        | Automatic via `meta/registry.json`    |
+| Update INDEX       | Manual edit after every page | Auto-rebuilds on turn end             |
+| Update LOG         | Manual append                | Auto-generated from `events.jsonl`    |
+| Find orphans       | Shell `grep` scans           | Instant from `backlinks.json`         |
+| Block raw edits    | Skill says "don't"           | Extension **enforces** immutability   |
+| Create source page | 8 tool calls                 | `wiki_capture_source` + LLM synthesis |
+
+## Available Tools
+
+Use these directly — they handle scaffolding and bookkeeping:
+
+- `wiki_bootstrap` — Initialize a new vault
+- `wiki_capture_source` — Capture URL/file/text into immutable packet + skeleton page
+- `wiki_ingest` — Get batch of uningested sources with extracted text
+- `wiki_ensure_page` — Create entity/concept/synthesis/analysis page from template
+- `wiki_search` — Search registry for existing pages
+- `wiki_lint` — Health check (orphans, missing, contradictions, gaps)
+- `wiki_status` — Instant stats
+- `wiki_rebuild_meta` — Force metadata rebuild
+- `wiki_log_event` — Record a custom event
+- `wiki_watch` — Schedule auto-updates
 
 ## Workflows
 
-### `/wiki-init` — Initialize a new wiki
-
-**When:** The user wants to start a new wiki on a topic.
-
-1. Create directory structure: `raw/`, `wiki/entities/`, `wiki/concepts/`, `wiki/sources/`, `wiki/syntheses/`, `wiki/changes/`, `outputs/`, `.discoveries/`
-2. Create `config.yaml` with the topic, any feeds, and default settings
-3. Create `wiki/INDEX.md` with placeholder sections
-4. Create `wiki/LOG.md` with initial entry
-5. Create `wiki/DASHBOARD.md` with Dataview queries
-6. Report the structure and suggest first steps
-
-### `/wiki-ingest [path]` — Process new sources
-
-**When:** New files appear in `raw/`. Process one or all new sources.
-
-1. Read `config.yaml` and this schema (for rules)
-2. Read `.discoveries/history.json` → get already-processed files
-3. Scan `raw/` → identify files not yet in history
-4. For each new source file:
-   a. Read the full content (use `read` for text, `fetch_content` for URLs, bash for PDF text extraction)
-   b. Discuss key takeaways with the user (1-2 sentence summary, ask if they want emphasis on anything)
-   c. Create a source summary page in `wiki/sources/`:
-   ```markdown
-   ---
-   type: source
-   format: article | paper | note | video | slack | podcast
-   raw_path: raw/articles/filename.md
-   ingested: YYYY-MM-DD
-   topics: [topic1, topic2]
-   ---
-
-   # Source Title
-
-   ## Summary
-
-   [2-3 paragraph summary of key content]
-
-   ## Key Takeaways
-
-   - [Bullet points of most important points]
-
-   ## Entities Mentioned
-
-   - [[entity-name]]
-
-   ## Concepts Mentioned
-
-   - [[concept-name]]
-
-   ## Notable Quotes
-
-   > "Quote" — attribution
-   ```
-   d. Create or update entity pages in `wiki/entities/`:
-   ```markdown
-   ---
-   type: entity
-   category: person | organization | tool | project | product
-   created: YYYY-MM-DD
-   updated: YYYY-MM-DD
-   sources: [raw/articles/filename.md]
-   ---
-
-   # Entity Name
-
-   One-line description.
-
-   ## Overview
-
-   [Key facts, role, significance]
-
-   ## Links
-
-   - [[related-concept]]
-   - [[related-entity]]
-
-   ## Sources
-
-   - [raw/articles/filename.md](../raw/articles/filename.md)
-   ```
-   e. Create or update concept pages in `wiki/concepts/`:
-   ```markdown
-   ---
-   type: concept
-   domain: ai | engineering | business | product | design | personal
-   created: YYYY-MM-DD
-   updated: YYYY-MM-DD
-   sources: [raw/articles/filename.md]
-   ---
-
-   # Concept Name
-
-   One-line definition.
-
-   ## Definition
-
-   [Clear explanation]
-
-   ## How It Works
-
-   [Technical details if applicable]
-
-   ## Examples
-
-   [Concrete examples from sources]
-
-   ## Links
-
-   - [[related-concept]]
-
-   ## Sources
-
-   - [raw/articles/filename.md](../raw/articles/filename.md)
-   ```
-   f. Add `[[wikilinks]]` cross-references to all related pages
-   g. Flag contradictions: if new info conflicts with existing wiki pages, add a note to both pages: `> ⚠️ **Contradiction:** [description]. See [[other-page]] and [source](../raw/path).`
-5. Update `wiki/INDEX.md` — add entries for every new or updated page
-6. Append to `wiki/LOG.md`: `## [YYYY-MM-DD HH:mm] ingest | [filename] | [N] pages affected`
-7. Update `.discoveries/history.json`
-8. Report summary: `Ingested [N] sources → [M] wiki pages created/updated. [X] contradictions flagged.`
-
-**Rules:**
-
-- One source can affect 5-15 wiki pages
-- Never fabricate — only write what's in the raw sources
-- If a file can't be read → log the error, skip it, continue
-- Batch size: process files one at a time for quality, or batch up to 5 for speed
+### Capture → Ingest → Synthesize
 
-### `/wiki-query <question>` — Ask questions against the wiki
+1. **Capture**: `wiki_capture_source(url="...")` → creates packet + skeleton
+2. **Ingest**: `wiki_ingest()` → get batch of sources needing synthesis
+3. **Read**: `read raw/sources/SRC-*/extracted.md`
+4. **Write**: Update skeleton source page with summary, entities, concepts
+5. **Ensure**: `wiki_ensure_page(type="entity", title="...")` for each entity
+6. **Cross-ref**: Add `[[wikilinks]]` between related pages
+7. **Done**: Extension auto-rebuilds metadata
 
-**When:** The user asks a question about wiki content.
+### Query → Answer → File
 
-1. Read `wiki/INDEX.md` to identify relevant pages
-2. Read those pages (get enough context — don't stop at 1-2 pages)
-3. Synthesize an answer with `[[wikilink]]` citations to specific pages
-4. If the answer is novel or analytically valuable → save it as a synthesis page in `wiki/syntheses/`
-5. Append to `wiki/LOG.md`
-6. Report the answer clearly with citations
-
-**Rules:**
-
-- Answer ONLY from wiki content, not from general knowledge
-- If the wiki lacks sufficient information → say so clearly, suggest what sources would help
-- Comparisons, analyses, and new connections → always save as synthesis pages (knowledge compounds)
-
-### `/wiki-lint` — Health check the wiki
-
-**When:** The user asks for a health check, or periodically.
-
-1. Read `wiki/INDEX.md` and scan all files in `wiki/`
-2. Check for:
-   - **Contradictions**: Claims that conflict between pages. Read related pages together to verify consistency.
-   - **Orphans**: Pages with zero inbound `[[wikilinks]]`
-   - **Missing pages**: `[[links]]` pointing to files that don't exist
-   - **Stale claims**: Information superseded by newer sources (check dates)
-   - **Broken raw links**: Links to `raw/` files that no longer exist
-   - **Knowledge gaps**: Important concepts mentioned but lacking their own page
-   - **Quality issues**: Pages under 3 lines, pages with no sources, pages with no cross-references
-3. Auto-fix: fix broken links, create missing pages for frequently-linked concepts, add cross-refs to orphans
-4. If contradictions found → flag them, don't silently resolve (ask the human which version is correct)
-5. Save report → `outputs/lint-YYYY-MM-DD.md`
-6. Update `.discoveries/gaps.json` with discovered gaps
-7. Append to `wiki/LOG.md`
-8. Report key findings
-
-### `/wiki-discover` — Auto-discover new sources
-
-**When:** The user wants to find new content for the wiki.
-
-1. Read `config.yaml` → extract topics, keywords, feeds, subreddits
-2. Read `.discoveries/gaps.json` → knowledge gaps identified by lint
-3. Read `.discoveries/history.json` → URLs already fetched
-4. Search for new sources using:
-   - **Web search**: Search topics + keywords from config
-   - **Gap filling**: Search for gaps identified in `.discoveries/gaps.json`
-   - **Trending**: Search for latest content on each topic
-5. For each promising source found:
-   a. Fetch the full content (use `fetch_content` or `web_search`)
-   b. Save to `raw/articles/YYYY-MM-DD-slug.md` with frontmatter:
-   ```markdown
-   ---
-   title: "Original Title"
-   url: "https://..."
-   discovered: YYYY-MM-DD
-   topic: "topic-name"
-   ---
-   ```
-6. Update `.discoveries/history.json`
-7. Report: `Discovered [N] new sources. Run /wiki-ingest to process them.`
-
-**Rules:**
-
-- Max 5-10 new sources per discover cycle
-- Skip: ads, shallow listicles, duplicates, paywalled content
-- Prefer: in-depth articles, papers, guides, high-quality analysis
-
-### `/wiki-run` — Full cycle: discover → ingest → lint
-
-**When:** The user wants a complete refresh.
-
-1. Run `discover` → find new sources
-2. Run `ingest` → process all new files in `raw/`
-3. Run `lint` → health check
-4. If lint found critical gaps → optionally run one more discover+ingest cycle
-5. Save summary report → `outputs/run-YYYY-MM-DD.md`
-6. Report final summary
-
-### `/wiki-status` — Show wiki health
-
-**When:** The user wants a quick overview.
-
-Report:
-
-```
-📊 LLM Wiki Status
-══════════════════
-Wiki Roots: [topic1], [topic2]
-Sources: [N] files
-Wiki Pages: [N] total ([E] entities, [C] concepts, [S] sources, [Y] syntheses)
-Last Ingest: YYYY-MM-DD
-Last Lint: YYYY-MM-DD
-Orphans: [N]
-Knowledge Gaps: [N]
-Health: ✅ Good | ⚠️ Warning | 🔴 Needs Attention
-```
-
-### `/wiki-digest` — Daily/Weekly digest
-
-**When:** The user wants a summary of recent changes.
-
-1. Read `wiki/LOG.md` — filter entries since last digest
-2. Summarize: new sources ingested, pages created/updated, insights discovered
-3. Report in a concise digest format
-4. Save → `outputs/digest-YYYY-MM-DD.md`
-
-### `/wiki-watch <interval>` — Auto-run on schedule
-
-**When:** The user wants the wiki to stay current automatically.
-
-Set up a recurring schedule using pi's schedule system:
-
-```
-/wiki-watch daily    → schedule_prompt action=add schedule="0 0 8 * * *" prompt="Run `/wiki-run` for the LLM Wiki"
-/wiki-watch weekly   → schedule_prompt action=add schedule="0 0 9 * * 1" prompt="Run `/wiki-run` for the LLM Wiki"
-/wiki-watch hourly   → schedule_prompt action=add schedule="0 0 * * * *" prompt="Run `/wiki-run` for the LLM Wiki"
-/wiki-watch stop     → list and remove scheduled wiki jobs
-```
-
----
+1. `wiki_search(query="...")` to find relevant pages
+2. Read those pages
+3. Synthesize answer with `[[wikilink]]` citations
+4. If novel: create analysis page via `wiki_ensure_page(type="analysis")`
+5. Extension auto-updates metadata
 
 ## Page Conventions
 
 ### Naming
 
-- File names: `kebab-case.md` (e.g., `attention-mechanism.md`, `elon-musk.md`)
-- Entity names: full name lowercased, hyphenated
-- Concept names: brief descriptive slug
+- `kebab-case.md` for all files
+- Folder-qualified wikilinks: `[[concepts/retrieval-augmented-generation]]`
 
 ### Frontmatter
 
-Every page MUST have YAML frontmatter:
-
 ```yaml
 ---
-type: entity | concept | source | synthesis | change
+type: entity | concept | source | synthesis | analysis
 created: YYYY-MM-DD
 updated: YYYY-MM-DD
-sources: [raw/path1.md, raw/path2.md]
+sources: [sources/SRC-YYYY-MM-DD-NNN]
 ---
 ```
 
-Entity pages also need: `category: person | organization | tool | project | product`
-Concept pages also need: `domain: ai | engineering | business | product | design | personal`
+Entity: add `category: person | organization | tool | project | product`
+Concept: add `domain: ai | engineering | business | product | design | personal`
 
-### Cross-references
+### Citations
 
-- Internal links: `[[kebab-case-file-name]]`
-- External links: `[text](url)`
-- Source citations: `[Source: filename](../raw/path/file.md)`
-- Every page needs ≥2 `[[wikilinks]]`
+Use stable source IDs: `[[sources/SRC-2026-04-28-001]]`
 
 ### Contradictions
 
-When sources disagree, document both sides:
-
 ```markdown
 > ⚠️ **Contradiction:** Source A claims X, but Source B claims Y.
-> See [[page-a]] and [[page-b]] for details.
->
-> - X: [Source A](../raw/path/a.md)
-> - Y: [Source B](../raw/path/b.md)
+> See [[page-a]] and [[page-b]].
 ```
 
----
-
 ## Variants
 
-### Personal Wiki (`config.yaml → wiki_mode: personal`)
-
-Track: learning, journaling, book notes, health, goals, reflections.
-
-- Extra folders: `wiki/journal/`, `wiki/goals/`
-- Personal entities: people you meet, books you read, courses you take
-- Daily journal entries can be ingested as raw notes
+### Personal Wiki
 
-### Company Wiki (`config.yaml → wiki_mode: company`)
+- Extra: `wiki/journal/`, `wiki/goals/`
+- Track: learning, books, health, reflections
 
-Track: competitors, market research, customer calls, internal docs, strategy.
+### Company Wiki
 
-- Extra folders: `wiki/changes/`, `wiki/decisions/`
-- Change detection: re-check competitor sources for pricing/feature changes
-- Slack/email threads as sources
-- Confidence levels in frontmatter: `confidence: high | medium | low`
-
----
+- Extra: `wiki/changes/`, `wiki/decisions/`
+- Track: competitors, market, strategy
+- Frontmatter: `confidence: high | medium | low`
 
 ## Obsidian Integration
 
-This wiki is designed to be opened as an Obsidian vault (`wiki/` directory).
-
-**Recommended Obsidian plugins:**
+Open `wiki/` as an Obsidian vault. The extension generates:
 
-- **Dataview** — Query pages by frontmatter (type, confidence, domain)
-- **Graph View** — Visualize `[[wikilink]]` connections
-- **Backlinks** — See what links to each page
-- **Spaced Repetition** — Flashcards from `wiki/flashcards.md`
-- **Charts View** — Dashboard analytics
-- **Marp** — Markdown slide decks
+- `meta/index.md` — browsable catalog
+- `meta/backlinks.json` — for graph plugins
+- `[[wikilinks]]` — native Obsidian links
 
-**The `wiki/DASHBOARD.md`** page includes Dataview queries that surface:
-
-- Low-confidence pages needing review
-- Recent updates
-- Concepts by domain
-- Pages with the most sources
-
----
-
-## Scheduling (Auto-Update)
-
-To keep the wiki always fresh, use pi's scheduling:
-
-```bash
-# Daily at 8 AM — discover + ingest + lint
-/wiki-watch daily
-
-# Or manually:
-/wiki-run
-```
-
-The schedule checks your configured topics, discovers new sources, ingests them, and lints for health — all automatically.
-
----
+Recommended plugins: Dataview, Graph View, Backlinks
 
 ## Tips
 
-- **Obsidian Web Clipper** — Browser extension that saves articles as markdown. Clip directly into `raw/articles/`.
-- **Download images**: In Obsidian Settings → Files & Links → set "Attachment folder path" to `raw/assets/`. After clipping, use "Download attachments" (Ctrl+Shift+D).
-- **Git versioning**: The wiki is a git repo. Every ingest is a commit. Revert bad ingests, review evolution.
-- **Start small**: 3-5 sources. Let the wiki grow organically. Don't try to ingest everything at once.
-- **Evolve the schema**: As you use the wiki, you'll find what works. Update rules here as you go.
+- **Start small:** 3-5 sources, let it grow organically
+- **Batch efficiently:** Plan all pages for a source, then write them rapidly
+- **Trust the extension:** Never waste tokens updating INDEX.md or LOG.md manually
+- **Evolve the schema:** Update `WIKI_SCHEMA.md` as conventions mature

package/skills/llm-wiki/templates/pages/analysis.md

@@ -0,0 +1,35 @@
+---
+type: analysis
+topic: ""
+created: YYYY-MM-DD
+updated: YYYY-MM-DD
+sources_count: 0
+---
+
+# Analysis Title
+
+> _Durable answer derived from wiki content._
+
+## Question
+
+[The original question that drove this analysis.]
+
+## Answer
+
+[Synthesized answer with [[wikilink]] citations to specific pages.]
+
+## Key Insights
+
+1. [Insight 1]
+2. [Insight 2]
+3. [Insight 3]
+
+## Sources Used
+
+- [[source-1]]
+- [[source-2]]
+
+## Related Pages
+
+- [[concept-1]]
+- [[entity-1]]

package/test/llm-wiki.test.ts

@@ -89,7 +89,7 @@ describe("package structure", () => {
     expect(content).toContain("## Golden Rules");
     expect(content).toContain("RAW IS IMMUTABLE");
     expect(content).toContain("## Workflows");
-    expect(content).toContain("/wiki-ingest");
+    expect(content).toContain("wiki_ingest");
     expect(content).toContain("Obsidian Integration");
     expect(content).toContain("Personal Wiki");
     expect(content).toContain("Company Wiki");
@@ -128,17 +128,28 @@ describe("package structure", () => {
     expect(existsSync(join(t, "pages", "synthesis.md"))).toBe(true);
   });
 
-  it("should have the extension file with all custom tools", () => {
-    const extPath = join(rootDir, "extensions", "llm-wiki-tools.ts");
+  it("should have the extension entry point", () => {
+    const extPath = join(rootDir, "extensions", "llm-wiki", "index.ts");
     expect(existsSync(extPath)).toBe(true);
     const content = readFile(extPath);
     expect(content).toContain("ExtensionAPI");
-    expect(content).toContain("registerTool");
+    expect(content).toContain("registerWikiBootstrap");
+  });
+
+  it("should have all custom tools in the extension", () => {
+    const toolsPath = join(rootDir, "extensions", "llm-wiki", "lib", "tools.ts");
+    expect(existsSync(toolsPath)).toBe(true);
+    const content = readFile(toolsPath);
     const tools = [
+      "wiki_bootstrap",
+      "wiki_capture_source",
       "wiki_ingest",
-      "wiki_status_report",
-      "wiki_lint_report",
-      "wiki_discover_sources",
+      "wiki_ensure_page",
+      "wiki_search",
+      "wiki_lint",
+      "wiki_status",
+      "wiki_rebuild_meta",
+      "wiki_log_event",
       "wiki_watch",
     ];
     for (const tool of tools) {
@@ -152,8 +163,6 @@ describe("package structure", () => {
     expect(readme).toContain("pi install npm:@zosmaai/pi-llm-wiki");
     expect(readme).toContain("Karpathy");
     expect(readme).toContain("Obsidian");
-    expect(readme).toContain("Personal Wiki");
-    expect(readme).toContain("Company Wiki");
   });
 });