@zosmaai/pi-llm-wiki 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-05-11
+
+### Added
+- **Overhauled README**: npm downloads badge, slash commands table, guardrails section, skill behavior, vault layout, source packet format, integration flow, linking style guide
+- **Better npm discoverability**: 19 keywords (was 10), expanded description with search terms, `files` field to slim package
+- **GitHub topics**: pi, llm-wiki, knowledge-base, wiki, markdown, obsidian, karpathy, second-brain, pkm, memory
+
+## [0.4.0] - 2026-05-11
+
+- JSON file support (PR #15 by jfraser)
+- Extractor strategy pattern refactor
+
+## [0.3.0] - 2026-05-07
+
+- Release
+
+## [0.2.2] - 2026-05-03
+
+- Fix: CodeQL alerts for safe tag stripping and entity decoding
+- Fix: README contributors via contrib.rocks
+- Added: Features section and env var documentation
+
+## [0.2.1] - 2026-04-29
+
+- Minor fixes
+
 ## [0.2.0] - 2026-04-28
 
 ### Added

package/README.md

@@ -2,53 +2,264 @@
 
 [![CI](https://github.com/zosmaai/pi-llm-wiki/actions/workflows/ci.yml/badge.svg)](https://github.com/zosmaai/pi-llm-wiki/actions/workflows/ci.yml)
 [![npm](https://img.shields.io/npm/v/@zosmaai/pi-llm-wiki)](https://www.npmjs.com/package/@zosmaai/pi-llm-wiki)
+[![npm downloads](https://img.shields.io/npm/dm/@zosmaai/pi-llm-wiki)](https://www.npmjs.com/package/@zosmaai/pi-llm-wiki)
 [![Coverage](https://codecov.io/gh/zosmaai/pi-llm-wiki/branch/main/graph/badge.svg)](https://codecov.io/gh/zosmaai/pi-llm-wiki)
 [![License: MIT](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
 [![CodeQL](https://github.com/zosmaai/pi-llm-wiki/actions/workflows/codeql.yml/badge.svg)](https://github.com/zosmaai/pi-llm-wiki/actions/workflows/codeql.yml)
 
-Self-maintaining, Obsidian-compatible knowledge base for [pi](https://pi.dev). Following Andrej Karpathy's LLM Wiki pattern.
+**Self-maintaining, Obsidian-compatible knowledge base for [pi](https://pi.dev).**
+Follows Andrej Karpathy's [LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f).
 
-## Install
+Turn raw sources (URLs, PDFs, markdown, JSON, XML) into a durable, interlinked, LLM-maintained wiki that compounds over time.
+
+---
+
+## Quick Start
 
 ```bash
 pi install npm:@zosmaai/pi-llm-wiki
 ```
 
-## Quick Start
-
 ```
 /wiki-init "AI Engineering"
+/wiki-ingest
+/wiki-query What are the key patterns?
 ```
 
-Drop sources into `raw/`, then:
+---
+
+## Why This Package?
+
+Most file-based LLM workflows behave like one-shot RAG: the model searches raw documents every time you ask a question. Synthesis is ephemeral.
+
+**pi-llm-wiki** creates a middle layer:
+
+- **Raw source packets** preserve source-of-truth inputs
+- **Source pages** summarize what each source says
+- **Canonical wiki pages** track what the wiki currently believes
+- **Generated metadata** keeps everything searchable and navigable
 
+The result is a wiki that **compounds** as you capture sources, ask questions, and file durable analyses.
+
+---
+
+## Features
+
+| Capability | Description |
+|------------|-------------|
+| 🔗 **Immutable source capture** | URLs, local files (PDF/md/txt/html/XML/JSON), or pasted text → structured source packets |
+| 🧠 **Automated ingestion** | `wiki_ingest` batch-processes sources into concept, entity, synthesis & analysis pages |
+| 🔍 **Full-text search** | Generated registry with keyword lookup across all pages and sources |
+| 🩺 **Mechanical linting** | Orphans, broken links, duplicate aliases, coverage gaps, stale captures |
+| 📊 **Dashboard** | `wiki_status` — counts, source states, recent activity |
+| 🤖 **Auto-update watch** | `wiki_watch` — schedule periodic discovery + ingest |
+| 📝 **Obsidian-friendly** | Folder-qualified wikilinks, stable source-ID citations, compatible vault |
+| 🛡️ **Guardrails** | Blocks direct edits to raw sources and generated metadata |
+| 🔧 **Configurable PDF extraction** | MarkItDown timeout via `WIKI_MARKITDOWN_TIMEOUT_MS` env var |
+| 🧪 **38+ tests, CI, CodeQL** | TypeScript, Vitest, Biome, Codecov |
+
+---
+
+## Tools
+
+| Tool | Description |
+|------|-------------|
+| `wiki_bootstrap` | Initialize a new wiki vault with config, templates, schema, and metadata |
+| `wiki_capture_source` | Capture a URL, local file, or pasted text into an immutable source packet |
+| `wiki_ingest` | Process uningested source packets into wiki pages (batch) |
+| `wiki_ensure_page` | Resolve or safely create entity / concept / synthesis / analysis pages |
+| `wiki_search` | Search the generated wiki registry |
+| `wiki_lint` | Deterministic health checks (orphans, gaps, contradictions, auto-fix) |
+| `wiki_status` | Show counts, source states, and recent activity |
+| `wiki_rebuild_meta` | Force a full metadata rebuild (registry, backlinks, index, log) |
+| `wiki_log_event` | Append a structured event to the wiki activity log |
+| `wiki_watch` | Schedule automatic wiki updates (daily / weekly / hourly) |
+
+### Slash Commands
+
+| Command | Description |
+|---------|-------------|
+| `/wiki-status` | Show a concise operational summary |
+| `/wiki-lint [mode]` | Run mechanical lint (`all`, `links`, `orphans`, `frontmatter`, `duplicates`, `coverage`, `staleness`) |
+| `/wiki-rebuild` | Force a full metadata rebuild |
+
+---
+
+## Quick Start (Detailed)
+
+### 1) Create a new wiki
+
+```bash
+mkdir my-wiki
+cd my-wiki
+pi
 ```
-/wiki-ingest
-/wiki-query What are the key patterns?
+
+Ask pi:
+
+```
+Initialize an llm wiki here for AI research.
 ```
 
-## What It Does
+This calls `wiki_bootstrap` and creates:
 
-| Tool                  | Purpose                                       |
-| --------------------- | --------------------------------------------- |
-| `wiki_bootstrap`      | Initialize a new wiki vault                   |
-| `wiki_capture_source` | Capture URL/file/text into immutable packet   |
-| `wiki_ingest`         | Process sources into wiki pages               |
-| `wiki_ensure_page`    | Create entity/concept/synthesis/analysis page |
-| `wiki_search`         | Search the wiki registry                      |
-| `wiki_lint`           | Health check (orphans, gaps, contradictions)  |
-| `wiki_status`         | Stats dashboard                               |
-| `wiki_rebuild_meta`   | Force metadata rebuild                        |
-| `wiki_log_event`      | Record custom event                           |
-| `wiki_watch`          | Schedule auto-updates                         |
+```
+raw/
+wiki/
+meta/
+.wiki/
+WIKI_SCHEMA.md
+```
 
-## Features
+### 2) Capture a source
 
-- **Configurable PDF extraction** — MarkItDown timeout adjustable via `WIKI_MARKITDOWN_TIMEOUT_MS` env var
-- **Smart content detection** — PDF bytes sniffed even from non-`.pdf` URLs, never written as markdown
-- **Original artifacts preserved** — URL captures save the fetched payload under `original/source.*`
-- **Clickable source links** — Captured URLs render as clickable Markdown links in source pages
-- **Reliable prompt forwarding** — Slash commands properly forward user arguments to the model
+```
+Capture this article into the wiki: https://example.com/some-article
+```
+
+```
+Capture this PDF into the wiki: ./papers/context-windows.pdf
+```
+
+```
+Capture these notes into the wiki: ...pasted text...
+```
+
+### 3) Integrate the source
+
+1. Capture the source
+2. Read `wiki/sources/SRC-*.md`
+3. Update that source page
+4. Search for impacted canonical pages with `wiki_search`
+5. Create missing pages with `wiki_ensure_page`
+6. Update concept / entity / synthesis pages with citations
+7. Mark the integration with `wiki_log_event kind=integrate`
+
+### 4) Query the wiki
+
+```
+Based on the wiki, what are the main tradeoffs between long-context models and RAG?
+```
+
+By default, query mode is **read-only**. To file a durable answer:
+
+```
+Answer the question and file the result as an analysis page.
+```
+
+---
+
+## Vault Layout
+
+```
+my-wiki/
+├─ raw/
+│  └─ sources/
+│     └─ SRC-2026-05-11-001/
+│        ├─ manifest.json
+│        ├─ original/           # Original artifact
+│        ├─ extracted.md        # Normalized text
+│        └─ attachments/
+├─ wiki/
+│  ├─ sources/                  # Source pages (what each source says)
+│  ├─ concepts/                 # Concepts and recurring ideas
+│  ├─ entities/                 # People, orgs, products, papers, systems
+│  ├─ syntheses/                # Cross-source theses and tensions
+│  └─ analyses/                 # Durable filed answers from queries
+├─ meta/
+│  ├─ registry.json             # Auto-generated search index
+│  ├─ backlinks.json
+│  ├─ index.md
+│  ├─ events.jsonl              # Append-only event log
+│  ├─ log.md
+│  └─ lint-report.md
+├─ .wiki/
+│  ├─ config.json
+│  └─ templates/
+└─ WIKI_SCHEMA.md
+```
+
+### Ownership Model
+
+| Path | Owner | Rule |
+|------|-------|------|
+| `raw/**` | Extension tools | Immutable after capture |
+| `wiki/**` | Model + user | Editable knowledge pages |
+| `meta/registry.json` | Extension | Generated |
+| `meta/backlinks.json` | Extension | Generated |
+| `meta/index.md` | Extension | Generated |
+| `meta/events.jsonl` | Extension / tool | Append-only |
+| `meta/log.md` | Extension | Generated from events |
+| `meta/lint-report.md` | Extension | Generated |
+| `WIKI_SCHEMA.md` | Human + explicit request | Operating manual |
+
+---
+
+## Linking & Citation Style
+
+### Internal Navigation
+
+```markdown
+[[concepts/retrieval-augmented-generation]]
+[[entities/openai|OpenAI]]
+[[syntheses/long-context-vs-rag]]
+```
+
+### Factual Citations
+
+```markdown
+[[sources/SRC-2026-04-04-001|SRC-2026-04-04-001]]
+```
+
+Stable source-page IDs keep provenance stable even if titles change.
+
+---
+
+## Guardrails
+
+The extension **blocks** direct tool-call edits to:
+
+- `raw/**` — immutable source artifacts
+- `meta/registry.json`
+- `meta/backlinks.json`
+- `meta/events.jsonl`
+- `meta/index.md`
+- `meta/log.md`
+- `meta/lint-report.md`
+
+If the model directly edits `wiki/**` using Pi's built-in `write` or `edit` tools, the extension **automatically rebuilds** generated metadata at the end of the agent turn.
+
+---
+
+## Source Packet Format
+
+Each captured source is stored as a structured packet:
+
+```
+raw/sources/SRC-YYYY-MM-DD-NNN/
+├─ manifest.json     # Capture metadata (title, URL, format, timestamp)
+├─ original/         # Original artifact (preserved as-is)
+├─ extracted.md      # Normalized text (PDF→md, XML→md, JSON→md, etc.)
+└─ attachments/      # Future attachment downloads
+```
+
+This preserves both the **original artifact** and a **normalized extracted view** for reading.
+
+---
+
+## Skill Behavior
+
+The bundled `llm-wiki` skill teaches the model to:
+
+- ❌ Never edit raw sources directly
+- ❌ Never edit generated metadata files
+- ✅ Capture first, integrate second
+- ✅ Search before creating new canonical pages
+- ✅ Cite facts using source-page IDs
+- ✅ Keep query mode read-only by default
+- ✅ Use "Tensions / caveats" and "Open questions" when evidence is mixed
+
+---
 
 ## Architecture
 
@@ -61,19 +272,27 @@ meta/                   # Auto-generated registry, backlinks, index, log
 .wiki/                  # Config and templates
 ```
 
-Read [docs/architecture.md](docs/architecture.md) for details.
+Read [docs/architecture.md](docs/architecture.md) for the full design document.
+
+---
 
 ## Documentation
 
-- [Architecture](docs/architecture.md) — How the four layers work
-- [Commands](docs/commands.md) — All slash commands and tools
-- [Obsidian Integration](docs/obsidian.md) — Vault setup and recommended plugins
-- [Configuration](docs/configuration.md) — Wiki modes, topics, settings
-- [API](docs/api.md) — Extension tool reference
+| Document | What it covers |
+|----------|---------------|
+| [Architecture](docs/architecture.md) | How the four layers work, ownership model |
+| [Commands](docs/commands.md) | All slash commands and tool reference |
+| [Obsidian Integration](docs/obsidian.md) | Vault setup and recommended plugins |
+| [Configuration](docs/configuration.md) | Wiki modes, topics, environment variables |
+| [API](docs/api.md) | Extension tool parameter reference |
+
+---
 
 ## Contributing
 
-See [CONTRIBUTING.md](CONTRIBUTING.md).
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, test patterns, and PR workflow.
+
+---
 
 ## Star History
 
@@ -88,7 +307,8 @@ See [CONTRIBUTING.md](CONTRIBUTING.md).
 ---
 
 <div align="center">
-  <sub>Built with ❤️ by <a href="https://github.com/zosmaai">zosmaai</a></sub>
+  <sub>Built with ❤️ by <a href="https://github.com/zosmaai">zosmaai</a> · </sub>
+  <a href="https://pi.dev">pi.dev</a> · <a href="https://github.com/zosmaai/pi-llm-wiki/issues">Issues</a>
 </div>
 
 ## License

package/package.json

@@ -1,17 +1,26 @@
 {
   "name": "@zosmaai/pi-llm-wiki",
-  "version": "0.4.0",
-  "description": "LLM Wiki for Pi — self-maintaining knowledge base following Karpathy's pattern. Obsidian-friendly, auto-updating, personal & company wiki.",
+  "version": "0.5.0",
+  "description": "Self-maintaining LLM Wiki for Pi — Karpathy-pattern knowledge base with immutable source capture, automated ingestion, search, linting, and Obsidian-compatible vault. auto-updating personal & company wiki.",
   "keywords": [
+    "pi",
     "pi-package",
-    "pi-skill",
     "pi-extension",
+    "pi-skill",
     "llm-wiki",
     "karpathy",
     "knowledge-base",
     "obsidian",
     "second-brain",
     "wiki",
+    "memory",
+    "markdown",
+    "pkm",
+    "personal-knowledge-management",
+    "research",
+    "vault",
+    "zettelkasten",
+    "rag",
     "zosmaai"
   ],
   "license": "MIT",
@@ -24,6 +33,19 @@
     "url": "https://github.com/zosmaai/pi-llm-wiki/issues"
   },
   "homepage": "https://github.com/zosmaai/pi-llm-wiki#readme",
+  "files": [
+    "extensions",
+    "skills",
+    "prompts",
+    "docs",
+    "assets",
+    "README.md",
+    "CHANGELOG.md",
+    "LICENSE"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
   "scripts": {
     "test": "vitest run",
     "test:watch": "vitest",

package/.coderabbit.yaml

@@ -1,43 +0,0 @@
-# CodeRabbit AI Code Review Configuration
-# Docs: https://docs.coderabbit.ai/guides/configure-coderabbit
-
-language: en-US
-early_access: false
-reviews:
-  profile: assertive
-  request_changes_workflow: true
-  high_level_summary: true
-  poem: false
-  review_status: true
-  collapse_walkthrough: false
-  auto_review:
-    enabled: true
-    ignore_title_keywords:
-      - "chore"
-      - "WIP"
-      - "wip"
-    drafts: false
-    base_branches:
-      - main
-  path_instructions:
-    - path: "extensions/**/*.ts"
-      instructions: |
-        Review TypeScript extension files for pi coding agent. Check that:
-        1. Tools use pi.registerTool() with TypeBox schemas
-        2. Error handling uses try/catch with proper messages
-        3. Unused parameters use underscore prefix or are removed
-        4. File exports a default function receiving ExtensionAPI
-    - path: "skills/**/SKILL.md"
-      instructions: |
-        Review SKILL.md for pi. Verify:
-        1. Frontmatter follows Agent Skills spec (name, description)
-        2. Workflows are clearly documented with steps
-        3. Golden rules and conventions are explicit
-    - path: "prompts/**/*.md"
-      instructions: |
-        Review prompt templates. Verify:
-        1. Frontmatter has description, args, section, topLevelCli
-        2. Instructions are actionable and clear
-
-chat:
-  auto_reply: true

package/.github/codeql/codeql-config.yml

@@ -1,12 +0,0 @@
-name: "CodeQL Config"
-
-queries:
-  - uses: security-and-quality
-
-# Suppress false positives for temporary file creation.
-# This is a single-user CLI tool — TOCTOU race conditions on temp directories
-# are not a valid threat model. The alerts only fire because tests pass
-# os.tmpdir() paths into production capture functions.
-query-filters:
-  - exclude:
-      id: js/insecure-temporary-file

package/.github/workflows/ci.yml

@@ -1,45 +0,0 @@
-name: CI
-
-on:
-  push:
-    branches: [main]
-  pull_request:
-    branches: [main]
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  quality:
-    runs-on: ubuntu-latest
-    strategy:
-      matrix:
-        node-version: [20, 22, 23, 24, 25]
-
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: ${{ matrix.node-version }}
-          cache: npm
-
-      - run: npm ci
-
-      - name: Type Check
-        run: npm run typecheck
-
-      - name: Lint
-        run: npm run lint
-
-      - name: Run Tests with Coverage
-        run: npm run test:coverage
-
-      - name: Upload Coverage to Codecov
-        if: ${{ matrix.node-version == 22 && github.event_name == 'push' }}
-        uses: codecov/codecov-action@v5
-        with:
-          files: ./coverage/lcov.info
-          fail_ci_if_error: false
-          verbose: true

package/.github/workflows/codeql.yml

@@ -1,39 +0,0 @@
-name: "CodeQL"
-
-on:
-  push:
-    branches: [main]
-  pull_request:
-    branches: [main]
-  schedule:
-    - cron: "0 6 * * 1" # Every Monday at 6 AM
-
-jobs:
-  analyze:
-    name: Analyze (${{ matrix.language }})
-    runs-on: ubuntu-latest
-    permissions:
-      security-events: write
-      packages: read
-      actions: read
-      contents: read
-
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - language: javascript-typescript
-            build-mode: none
-
-    steps:
-      - uses: actions/checkout@v4
-
-      - uses: github/codeql-action/init@v3
-        with:
-          languages: ${{ matrix.language }}
-          build-mode: ${{ matrix.build-mode }}
-          config-file: ./.github/codeql/codeql-config.yml
-
-      - uses: github/codeql-action/analyze@v3
-        with:
-          category: "/language:${{ matrix.language }}"

package/.github/workflows/release.yml

@@ -1,86 +0,0 @@
-name: Release
-
-on:
-  push:
-    tags:
-      - "v*"
-
-jobs:
-  release:
-    runs-on: ubuntu-latest
-    permissions:
-      contents: write
-      id-token: write
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: 22
-          registry-url: https://registry.npmjs.org
-          cache: npm
-
-      - run: npm ci
-
-      - name: Type Check
-        run: npm run typecheck
-
-      - name: Lint
-        run: npm run lint
-
-      - name: Run Tests
-        run: npm test
-
-      - name: Extract Version from Tag
-        id: version
-        run: echo "VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_OUTPUT
-
-      - name: Update package.json version
-        run: |
-          node -e "const p=require('./package.json'); p.version='${{ steps.version.outputs.VERSION }}'; require('fs').writeFileSync('./package.json', JSON.stringify(p, null, 2)+'\n')"
-          npx biome check --fix package.json 2>&1
-
-      - name: Commit version bump back to main
-        run: |
-          git config user.name "github-actions[bot]"
-          git config user.email "github-actions[bot]@users.noreply.github.com"
-          git add package.json
-          git commit -m "chore(release): v${{ steps.version.outputs.VERSION }}"
-          git push origin HEAD:main
-
-      - name: Generate Release Notes
-        id: notes
-        run: |
-          CHANGELOG=$(git log --pretty=format:"- %s (%h)" $(git describe --tags --abbrev=0 HEAD~1 2>/dev/null || git rev-list --max-parents=0 HEAD)..HEAD -- . 2>/dev/null || echo "- Initial release")
-          echo "NOTES<<EOF" >> $GITHUB_OUTPUT
-          echo "$CHANGELOG" >> $GITHUB_OUTPUT
-          echo "EOF" >> $GITHUB_OUTPUT
-
-      - name: Publish to npm
-        run: npm publish --provenance --access public
-        env:
-          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
-
-      - name: Create GitHub Release
-        uses: softprops/action-gh-release@v2
-        with:
-          name: "v${{ steps.version.outputs.VERSION }}"
-          body: |
-            ## 📦 @zosmaai/pi-llm-wiki v${{ steps.version.outputs.VERSION }}
-
-            ${{ steps.notes.outputs.NOTES }}
-
-            ### Install
-            \`\`\`bash
-            pi install npm:@zosmaai/pi-llm-wiki@${{ steps.version.outputs.VERSION }}
-            \`\`\`
-
-            ### What's Included
-            - **Skill**: Self-maintaining wiki schema (SKILL.md)
-            - **Extension**: 10 TypeScript tools (wiki_bootstrap, wiki_capture_source, wiki_ingest, wiki_ensure_page, wiki_search, wiki_lint, wiki_status, wiki_rebuild_meta, wiki_log_event, wiki_watch)
-            - **Prompt Templates**: 8 slash commands (/wiki:init, :ingest, :query, :lint, :discover, :run, :status, :digest)
-            - **Templates**: INDEX.md, LOG.md, DASHBOARD.md, config.yaml, 5 page templates
-          generate_release_notes: false