@zosmaai/pi-llm-wiki 0.8.0 → 0.8.2

package/README.md

@@ -120,6 +120,7 @@ The result is a wiki that **compounds** as you capture sources, ask questions, a
 | `/wiki-status` | Show a concise operational summary |
 | `/wiki-digest [--period daily\|weekly]` | Generate a digest of recent activity |
 | `/wiki-retro` | Save atomic insights from completed tasks |
+| `/wiki-req <concept>` | Decompose a concept into atomic, traceable requirement pages |
 
 ---
 
@@ -398,9 +399,39 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup, test patterns, and
 
 ## Contributors
 
-<a href="https://github.com/zosmaai/pi-llm-wiki/graphs/contributors">
-  <img src="https://contrib.rocks/image?repo=zosmaai/pi-llm-wiki" alt="Contributors" />
-</a>
+Thanks to everyone who has contributed! This list is regenerated automatically by [`.github/workflows/contributors.yml`](.github/workflows/contributors.yml) — see [#60](https://github.com/zosmaai/pi-llm-wiki/issues/60) for the rationale.
+
+<!-- readme: contributors -start -->
+<table>
+	<tbody>
+		<tr>
+            <td align="center">
+                <a href="https://github.com/arjun-zosma">
+                    <img src="https://avatars.githubusercontent.com/u/25246034?v=4" width="64;" alt="arjun-zosma"/>
+                    <br />
+                    <sub><b>Arjun Nayak</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/jfraser">
+                    <img src="https://avatars.githubusercontent.com/u/165964?v=4" width="64;" alt="jfraser"/>
+                    <br />
+                    <sub><b>James Fraser</b></sub>
+                </a>
+            </td>
+            <td align="center">
+                <a href="https://github.com/Shanvit7">
+                    <img src="https://avatars.githubusercontent.com/u/64424817?v=4" width="64;" alt="Shanvit7"/>
+                    <br />
+                    <sub><b>Shanvit S Shetty</b></sub>
+                </a>
+            </td>
+		</tr>
+	<tbody>
+</table>
+<!-- readme: contributors -end -->
+
+<sub>Full history: [contributors graph](https://github.com/zosmaai/pi-llm-wiki/graphs/contributors).</sub>
 
 ---
 

package/docs/api.md

@@ -1,105 +1,353 @@
 # API Reference
 
-## Extension Tools
+All 13 tools registered by the extension. Parameters marked `?` are optional.
 
-### wiki_bootstrap
+---
 
-Initialize a new LLM Wiki vault.
+## wiki_bootstrap
+
+Initialize a new LLM Wiki vault with the 4-layer architecture. Creates config, templates, schema,
+and metadata scaffolding.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `topic` | `string` | ✅ | Main topic of the wiki |
+| `mode` | `string` | — | `"personal"` or `"company"` (default: `"personal"`) |
+| `root` | `string` | — | Root directory to bootstrap in (default: current working directory) |
+
+**Returns**
 
 ```
-wiki_bootstrap(topic: string, mode?: "personal" | "company", root?: string)
+details: { root: string, mode: string, topic: string }
 ```
 
-### wiki_capture_source
+Confirmation text includes the vault path, directory layout, and a prompt to capture the first source.
+
+---
+
+## wiki_capture_source
 
-Capture a URL, file, or text into an immutable source packet.
+Capture a URL, local file, or pasted text into an immutable source packet and skeleton source page.
+Provide exactly one of `url`, `file_path`, or `text`.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `url` | `string` | — | URL to fetch and capture |
+| `file_path` | `string` | — | Absolute or relative path to a local file (PDF, md, txt, html, XML, JSON) |
+| `text` | `string` | — | Pasted text content to capture directly |
+| `title` | `string` | — | Title override (used for `text` captures; inferred from URL/file otherwise) |
+
+**Returns**
 
 ```
-wiki_capture_source(
-  inputType: "url" | "file" | "text",
-  value: string,
-  title?: string,
-  kind?: string,
-  tags?: string[],
-  createSourcePage?: boolean
-)
+details: {
+  sourceId: string,          // e.g. "SRC-2026-06-03-001"
+  packetPath: string,        // path to raw/sources/SRC-.../
+  sourcePagePath: string,    // path to wiki/sources/SRC-....md (skeleton)
+  extractedPreview: string   // first 300 chars of extracted content
+}
 ```
 
-### wiki_ingest
+Errors with `isError: true` if no vault exists or no source input is provided.
+
+---
+
+## wiki_ingest
+
+Return a batch of uningested source packets for the LLM to synthesize. Does not write anything
+itself — the model reads the returned extracted content, fills in the skeleton source page,
+and creates entity/concept pages.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `source_id` | `string` | — | Process a specific source ID only; leave empty to get the next unprocessed batch |
+| `batch_size` | `number` | — | Max sources to return (default: `3`, max: `5`) |
+
+**Returns**
+
+```
+details: {
+  batch: string[],    // source IDs in this batch, e.g. ["SRC-2026-06-03-001"]
+  remaining: number   // sources still waiting after this batch
+}
+```
+
+Each batch entry includes the source title, char count, and the path to read (`raw/sources/{id}/extracted.md`).
+Returns a "all sources ingested" message with `{ ingested, total }` when nothing is pending.
+
+---
+
+## wiki_ensure_page
+
+Resolve or safely create a canonical wiki page. Returns immediately if the page already exists
+(no overwrite). Uses a built-in template when `content` is not provided.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `type` | `string` | ✅ | Page type: `"entity"`, `"concept"`, `"synthesis"`, `"analysis"`, or `"requirement"` |
+| `title` | `string` | ✅ | Human-readable page title; auto-slugified to a kebab-case filename |
+| `content` | `string` | — | Full markdown content for the page; if omitted, the type-appropriate template is used |
+
+**Returns**
+
+```
+details: { path: string, created: boolean }
+```
+
+`created: false` means the page already existed and was not modified.
+
+---
+
+## wiki_recall
+
+Search both the personal (`~/.llm-wiki/`) and project (`.llm-wiki/`) vaults for pages relevant to
+a query. Uses chunk-level scoring, weighted field matching, and pseudo-relevance feedback. Also
+called automatically before every agent turn.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `query` | `string` | ✅ | Search query — use the user's full request or key terms |
+| `max_results` | `number` | — | Maximum pages to return (default: `5`, max: `10`) |
+
+**Returns**
+
+```
+details: {
+  query: string,
+  matches: Array<{
+    id: string,           // folder-qualified page ID, e.g. "concepts/rag"
+    title: string,
+    type: string,         // "source" | "entity" | "concept" | "synthesis" | "analysis"
+    preview: string,      // best-matching chunk or page intro (~200 chars)
+    path: string,         // absolute filesystem path to the .md file
+    score: number,        // relevance score (higher = better)
+    vaultLabel?: string   // "📓 personal" when result is from the personal vault
+  }>
+}
+```
+
+Returns empty `matches: []` with a hint to use `wiki_retro` when the wiki has no matching pages.
+
+---
+
+## wiki_search
 
-Get a batch of sources needing synthesis.
+Exact keyword search across the generated registry. Faster and simpler than `wiki_recall` — no
+scoring, no PRF, no vault layering. Use for lookups when you already know what you're looking for.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `query` | `string` | ✅ | Search term matched against page IDs, titles, and types |
+| `type` | `string` | — | Filter results to a specific page type (e.g. `"concept"`, `"entity"`) |
+
+**Returns**
 
 ```
-wiki_ingest(batch_size?: number)
+details: {
+  query: string,
+  matches: Array<{ id: string, title: string, type: string }>
+}
 ```
 
-### wiki_ensure_page
+---
+
+## wiki_retro
+
+Save an atomic insight from a completed task as a single lightweight markdown file in
+`wiki/sources/`. Does not create a full source packet. Rebuilds metadata immediately so the
+insight is searchable in the same session.
+
+**Parameters**
 
-Create or resolve a canonical page.
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `slug` | `string` | ✅ | Unique kebab-case identifier (e.g. `"jwt-revocation-pattern"`). Used as the filename and for lookups. |
+| `title` | `string` | ✅ | Short descriptive title, 60 chars max. Noun phrase, not a sentence. |
+| `body` | `string` | ✅ | Markdown content explaining what was learned. Include `[[wikilinks]]` to related pages. |
+| `category` | `string` | — | Optional grouping label (e.g. `"frontend"`, `"architecture"`, `"devops"`, `"bugfix"`) |
+
+**Returns**
 
 ```
-wiki_ensure_page(
-  type: "concept" | "entity" | "synthesis" | "analysis",
-  title: string,
-  aliases?: string[],
-  tags?: string[],
-  summary?: string,
-  createIfMissing?: boolean
-)
+details: { slug: string, title: string, category: string | null }
 ```
 
-### wiki_search
+---
+
+## wiki_observe
+
+Record a timestamped, relevance-rated observation during a session. Saved to `wiki/sources/` with
+`status: observation`. Immediately searchable via `wiki_recall`. Intended for mid-session capture;
+use `wiki_retro` for end-of-task summaries.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `title` | `string` | ✅ | Short descriptive title, ≤80 chars. Noun phrase, not a sentence. |
+| `content` | `string` | ✅ | Plain prose: what happened, was decided, or was learned. Preserve specifics (file paths, function names, error messages, numbers). |
+| `relevance` | `"low" \| "medium" \| "high" \| "critical"` | ✅ | Retention priority. `low` = routine; `medium` = task context; `high` = non-trivial decisions; `critical` = persistent identity/preference or completed work that must not be redone. |
+| `tags` | `string` | — | Space-separated tags for categorisation (e.g. `"auth backend migration"`) |
+| `source_context` | `string` | — | What was being worked on (e.g. `"Adding authentication module"`) |
 
-Search the wiki registry.
+**Returns**
 
 ```
-wiki_search(query: string, type?: string, limit?: number)
+details: { slug: string, title: string, relevance: string, tags: string | null }
 ```
 
-### wiki_lint
+The slug is auto-generated as `obs-YYYY-MM-DD-{title-slug}`.
 
-Health check the wiki.
+---
+
+## wiki_lint
+
+Deterministic health check of the wiki. Scans for orphan pages (no inbound links), missing pages
+(linked but not created), and contradiction markers. Optionally auto-creates stub pages for
+knowledge gaps cited in two or more pages.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `auto_fix` | `boolean` | — | When `true`, auto-creates stub concept pages for gaps mentioned in ≥2 pages (default: `false`) |
+
+**Returns**
 
 ```
-wiki_lint(mode?: string, writeReport?: boolean, limit?: number)
+details: {
+  pages: number,
+  orphans: number,
+  missingPages: number,
+  contradictions: number,
+  reportPath: string,   // path to the generated lint report .md file
+  gaps: number          // knowledge gaps tracked in .discoveries/gaps.json
+}
 ```
 
-### wiki_status
+The lint report is written to `.llm-wiki/outputs/lint-YYYY-MM-DD.md`.
+Contradictions are flagged by the presence of `⚠️ **Contradiction` markers in page content and
+always require human review.
+
+---
+
+## wiki_status
+
+Report wiki health and statistics from the generated registry. Reads pre-built metadata — does not
+scan files directly.
+
+**Parameters**
+
+None.
 
-Show wiki statistics.
+**Returns**
 
 ```
-wiki_status()
+details: {
+  topic: string,
+  mode: string,               // "personal" or "company"
+  totalPages: number,
+  byType: Record<string, number>,  // e.g. { concept: 4, entity: 2, source: 7 }
+  orphans: number,
+  gaps: number,
+  health: "✅ Good" | "⚠️ Warning" | "🔴 Empty"
+}
 ```
 
-### wiki_rebuild_meta
+Health is `"⚠️ Warning"` when orphan count exceeds 5, `"🔴 Empty"` when the registry has no pages.
 
-Force metadata rebuild.
+---
+
+## wiki_rebuild_meta
+
+Force a full synchronous rebuild of all generated metadata: `registry.json`, `backlinks.json`,
+`index.md`, `log.md`. Use when metadata appears out of sync with actual wiki files.
+
+**Parameters**
+
+None.
+
+**Returns**
 
 ```
-wiki_rebuild_meta()
+details: { pageCount: number }
 ```
 
-### wiki_log_event
+---
 
-Record a structured event.
+## wiki_log_event
+
+Append a structured event to `meta/events.jsonl` and regenerate `meta/log.md`. Every event is
+timestamped automatically.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `kind` | `string` | ✅ | Event kind label (e.g. `"ingest"`, `"query"`, `"decision"`, `"integrate"`) |
+| `details` | `object` | — | Arbitrary additional fields to store alongside the event |
+
+**Returns**
 
 ```
-wiki_log_event(
-  kind: string,
-  title: string,
-  summary?: string,
-  sourceIds?: string[],
-  pagePaths?: string[],
-  notes?: string[],
-  actor?: "agent" | "user" | "extension"
-)
+details: { kind: string }
 ```
 
-### wiki_watch
+---
+
+## wiki_watch
 
-Schedule auto-updates.
+Output the shell command needed to schedule automatic wiki updates (discover → ingest → lint) via
+pi's `schedule_prompt` cron system. Does not schedule anything directly — it returns the command
+for the user to run.
+
+**Parameters**
+
+| Name | Type | Required | Description |
+|------|------|----------|-------------|
+| `interval` | `string` | ✅ | `"daily"` (8:00 AM), `"weekly"` (Monday 9:00 AM), `"hourly"`, or `"stop"` (prints removal instructions) |
+
+**Returns**
 
 ```
-wiki_watch(schedule: string, command: string)
+details: {
+  interval: string,
+  cronSchedule: string,   // e.g. "0 0 8 * * *"
+  label: string           // e.g. "Daily at 8:00 AM"
+}
+```
+
+When `interval` is `"stop"`, returns `details: { action: "stop_instructions" }` with instructions
+for removing existing jobs via `schedule_prompt action=remove`.
+
+---
+
+## Error Shape
+
+All tools return `isError: true` in their result when a hard error occurs (no vault found, missing
+required input). The `text` content will contain a human-readable explanation. Check for `isError`
+before using `details`.
+
+```ts
+{
+  content: [{ type: "text", text: string }],
+  details: { error: string },
+  isError: true
+}
 ```
+
+The most common error is **"No wiki found — run wiki_bootstrap first"**, returned by every tool
+except `wiki_bootstrap` itself when `.llm-wiki/config.json` does not exist in the resolved vault
+root.

package/extensions/llm-wiki/lib/source-extractors.ts

@@ -1,14 +1,23 @@
+import { open } from "node:fs/promises";
 import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
+import { NodeHtmlMarkdown } from "node-html-markdown";
 import { exec } from "./utils.js";
 
+export type ExtractionStatus = "success" | "failed" | "unsupported";
+
 export interface ExtractedContent {
   extracted: string;
   title?: string;
+  extractor?: string;
+  extraction_status?: ExtractionStatus;
+  content_type?: string;
 }
 
 export interface FileExtractor {
   format: string;
   shouldReadText: boolean;
+  extractorName?: string;
+  content_type?: string;
   matches(filePath: string): boolean;
   extract(args: FileExtractArgs): Promise<string> | string;
 }
@@ -31,6 +40,67 @@ interface UrlExtractArgs {
   signal?: AbortSignal;
 }
 
+// ---------------------------------------------------------------------------
+// Binary magic byte detection
+// ---------------------------------------------------------------------------
+
+const BINARY_SIGNATURES: Array<{ bytes: number[]; format: string }> = [
+  // Archives & documents
+  { bytes: [0x50, 0x4b, 0x03, 0x04], format: "zip" }, // ZIP / DOCX / XLSX / PPTX / JAR
+  { bytes: [0x25, 0x50, 0x44, 0x46], format: "pdf" }, // %PDF
+  { bytes: [0x37, 0x7a, 0xbc, 0xaf], format: "7z" }, // 7-Zip
+  { bytes: [0x1f, 0x8b], format: "gzip" }, // gzip / .tar.gz
+  // Images
+  { bytes: [0x89, 0x50, 0x4e, 0x47], format: "png" }, // PNG
+  { bytes: [0xff, 0xd8, 0xff], format: "jpeg" }, // JPEG
+  { bytes: [0x47, 0x49, 0x46, 0x38], format: "gif" }, // GIF8
+  { bytes: [0x42, 0x4d], format: "bmp" }, // BMP
+  { bytes: [0x49, 0x49, 0x2a, 0x00], format: "tiff" }, // TIFF (little-endian)
+  { bytes: [0x4d, 0x4d, 0x00, 0x2a], format: "tiff" }, // TIFF (big-endian)
+  { bytes: [0x52, 0x49, 0x46, 0x46], format: "riff" }, // RIFF (WAV / AVI / WebP)
+  // Executables & binaries
+  { bytes: [0x4d, 0x5a], format: "exe" }, // Windows PE (EXE / DLL)
+  { bytes: [0xcf, 0xfa, 0xed, 0xfe], format: "macho" }, // Mach-O 64-bit LE
+  { bytes: [0xce, 0xfa, 0xed, 0xfe], format: "macho" }, // Mach-O 32-bit LE
+  { bytes: [0xfe, 0xed, 0xfa, 0xcf], format: "macho" }, // Mach-O 64-bit BE
+  { bytes: [0xfe, 0xed, 0xfa, 0xce], format: "macho" }, // Mach-O 32-bit BE
+  { bytes: [0xca, 0xfe, 0xba, 0xbe], format: "class" }, // Java .class / Mach-O FAT
+  { bytes: [0x7f, 0x45, 0x4c, 0x46], format: "elf" }, // ELF binary
+  { bytes: [0x00, 0x61, 0x73, 0x6d], format: "wasm" }, // WebAssembly
+  // Data & media
+  { bytes: [0x53, 0x51, 0x4c, 0x69], format: "sqlite" }, // SQLite
+  { bytes: [0x49, 0x44, 0x33], format: "mp3" }, // MP3 (ID3 tag)
+];
+
+/**
+ * Reads the first 8 bytes of `filePath` and checks them against known binary
+ * magic byte signatures. Returns the detected format name or `null` for text.
+ */
+export async function detectBinaryMagicBytes(filePath: string): Promise<string | null> {
+  let handle: import("node:fs/promises").FileHandle | undefined;
+  try {
+    handle = await open(filePath, "r");
+    const buf = Buffer.alloc(8);
+    const { bytesRead } = await handle.read(buf, 0, 8, 0);
+    const header = buf.subarray(0, bytesRead);
+
+    for (const { bytes, format } of BINARY_SIGNATURES) {
+      if (bytes.every((b, i) => header[i] === b)) return format;
+    }
+    return null;
+  } catch {
+    return null; // Unreadable file — let the extractor deal with it
+  } finally {
+    await handle?.close();
+  }
+}
+
+export function binaryExtractionFailureMessage(format: string): string {
+  return `_Binary file could not be converted to markdown (detected format: ${format}).\nCapture a text-based version or a URL pointing to readable content instead._\n`;
+}
+
+// ---------------------------------------------------------------------------
+
 const DEFAULT_MARKITDOWN_TIMEOUT_MS = 180_000;
 const DEFAULT_CURL_TIMEOUT_SECONDS = 30;
 
@@ -38,25 +108,38 @@ const FILE_EXTRACTORS: FileExtractor[] = [
   {
     format: "pdf",
     shouldReadText: false,
+    extractorName: "markitdown",
+    content_type: "application/pdf",
     matches: hasExtension(".pdf"),
     extract: ({ pi, filePath, signal }) => extractPdf(pi, filePath, signal),
   },
-  textFileExtractor("markdown", [".md"]),
-  textFileExtractor("text", [".txt"]),
-  textFileExtractor("html", [".html", ".htm"]),
+  textFileExtractor("markdown", [".md"], "text/markdown"),
+  textFileExtractor("text", [".txt"], "text/plain"),
+  textFileExtractor("html", [".html", ".htm"], "text/html"),
   {
     format: "xml",
     shouldReadText: true,
+    extractorName: "xmlToMarkdown",
+    content_type: "application/xml",
     matches: hasExtension(".xml"),
     extract: ({ content }) => xmlToMarkdown(content),
   },
   {
     format: "json",
     shouldReadText: true,
+    extractorName: "jsonToMarkdown",
+    content_type: "application/json",
     matches: hasExtension(".json"),
     extract: ({ content }) => jsonToMarkdown(content),
   },
-  textFileExtractor("docx", [".docx"]),
+  {
+    format: "docx",
+    shouldReadText: false,
+    extractorName: "markitdown",
+    content_type: "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+    matches: hasExtension(".docx"),
+    extract: ({ pi, filePath, signal }) => extractDocx(pi, filePath, signal),
+  },
   textFileExtractor("file", []),
 ];
 
@@ -91,10 +174,16 @@ export function pdfExtractionFailureMessage(source: string): string {
   return `_PDF content could not be converted to markdown from ${source}. Try increasing WIKI_MARKITDOWN_TIMEOUT_MS._\n`;
 }
 
-function textFileExtractor(format: string, extensions: string[]): FileExtractor {
+function textFileExtractor(
+  format: string,
+  extensions: string[],
+  contentType?: string,
+): FileExtractor {
   return {
     format,
     shouldReadText: true,
+    extractorName: "passthrough",
+    content_type: contentType,
     matches: extensions.length ? hasAnyExtension(extensions) : () => true,
     extract: ({ content }) => content,
   };
@@ -113,13 +202,33 @@ async function extractPdf(pi: ExtensionAPI, source: string, signal?: AbortSignal
   return extracted || pdfExtractionFailureMessage(source);
 }
 
+export function docxExtractionFailureMessage(source: string): string {
+  return `_DOCX content could not be converted to markdown from ${source}. Ensure uvx and markitdown are installed._\n`;
+}
+
+async function extractDocx(
+  pi: ExtensionAPI,
+  source: string,
+  signal?: AbortSignal,
+): Promise<string> {
+  const extracted = await extractWithMarkItDown(pi, source, signal);
+  return extracted || docxExtractionFailureMessage(source);
+}
+
 async function extractPdfUrl(
   pi: ExtensionAPI,
   url: string,
   signal?: AbortSignal,
 ): Promise<ExtractedContent> {
   const extracted = await extractPdf(pi, url, signal);
-  return { extracted, title: titleFromMarkdown(extracted) };
+  const failed = extracted.includes("could not be converted");
+  return {
+    extracted,
+    title: titleFromMarkdown(extracted),
+    extractor: "markitdown",
+    extraction_status: failed ? "failed" : "success",
+    content_type: "application/pdf",
+  };
 }
 
 async function extractTextUrl(
@@ -129,13 +238,31 @@ async function extractTextUrl(
 ): Promise<ExtractedContent> {
   const markitdownExtracted = await extractWithMarkItDown(pi, url, signal);
   if (markitdownExtracted) {
-    return { extracted: markitdownExtracted, title: titleFromMarkdown(markitdownExtracted) };
+    return {
+      extracted: markitdownExtracted,
+      title: titleFromMarkdown(markitdownExtracted),
+      extractor: "markitdown",
+      extraction_status: "success",
+    };
   }
 
   const curlExtracted = await fetchTextUrl(pi, url, signal);
-  if (!curlExtracted) return { extracted: "" };
-  if (looksLikePdf(curlExtracted)) return { extracted: pdfExtractionFailureMessage(url) };
-  return { extracted: curlExtracted, title: titleFromHtml(curlExtracted) };
+  if (!curlExtracted) return { extracted: "", extractor: "none", extraction_status: "failed" };
+  if (looksLikePdf(curlExtracted)) {
+    return {
+      extracted: pdfExtractionFailureMessage(url),
+      extractor: "curl",
+      extraction_status: "failed",
+      content_type: "application/pdf",
+    };
+  }
+  const normalized = htmlToMarkdown(curlExtracted);
+  return {
+    extracted: normalized,
+    title: titleFromMarkdown(normalized) ?? titleFromHtml(curlExtracted),
+    extractor: "htmlToMarkdown",
+    extraction_status: "success",
+  };
 }
 
 async function extractWithMarkItDown(
@@ -149,7 +276,7 @@ async function extractWithMarkItDown(
     const mdResult = await exec(
       pi,
       "sh",
-      ["-c", `uvx --from 'markitdown[pdf]' markitdown "${source}" 2>/dev/null || echo ""`],
+      ["-c", `uvx --from 'markitdown[docx,pdf]' markitdown "${source}" 2>/dev/null || echo ""`],
       { signal, timeout: markitdownTimeoutMs() },
     );
     return mdResult.stdout.trim() ? mdResult.stdout : "";
@@ -216,6 +343,23 @@ function titleFromHtml(html: string): string | undefined {
   return html.match(/<title>([^<]*)<\/title>/i)?.[1]?.trim();
 }
 
+/** Decode common HTML/XML entities. Shared by xmlToMarkdown and htmlToMarkdown. */
+function decodeHtmlEntities(text: string): string {
+  return text.replace(/&(?:amp|lt|gt|quot|apos|#\d+);/gi, (entity) => {
+    const map: Record<string, string> = {
+      "&amp;": "&",
+      "&lt;": "<",
+      "&gt;": ">",
+      "&quot;": '"',
+      "&apos;": "'",
+    };
+    const lower = entity.toLowerCase();
+    if (map[lower]) return map[lower];
+    if (lower.startsWith("&#")) return String.fromCodePoint(Number.parseInt(entity.slice(2, -1)));
+    return entity;
+  });
+}
+
 /** Basic XML to markdown conversion: strip tags while preserving text structure. */
 function xmlToMarkdown(xml: string): string {
   let title = "";
@@ -234,14 +378,7 @@ function xmlToMarkdown(xml: string): string {
   }
   text = text.replace(/</g, "");
 
-  text = text.replace(/&(?:amp|lt|gt|quot|#\d+);/gi, (entity) => {
-    const map: Record<string, string> = { "&amp;": "&", "&lt;": "<", "&gt;": ">", "&quot;": '"' };
-    const lower = entity.toLowerCase();
-    if (map[lower]) return map[lower];
-    if (lower.startsWith("&#")) return String.fromCodePoint(Number.parseInt(entity.slice(2, -1)));
-    return entity;
-  });
-
+  text = decodeHtmlEntities(text);
   text = text.replace(/\n{3,}/g, "\n\n").trim();
   if (!text) return xml;
 
@@ -251,6 +388,41 @@ function xmlToMarkdown(xml: string): string {
   return lines.join("\n\n");
 }
 
+/**
+ * Lightweight HTML-to-markdown normalizer for the curl fallback path.
+ *
+ * Pre-strips page chrome (nav, header, footer, script, style) that
+ * node-html-markdown does not remove, then delegates full conversion —
+ * bold, italic, code blocks, tables, ordered lists, image alt text — to
+ * node-html-markdown. Prepends the <title> as a # heading when the body
+ * has no <h1> of its own.
+ *
+ * Falls back to the original HTML if conversion yields an empty string.
+ */
+export function htmlToMarkdown(input: string): string {
+  // 1. Extract <title> from original before stripping head
+  const title = input.match(/<title[^>]*>([^<]*)<\/title>/i)?.[1]?.trim() ?? "";
+
+  // 2. Strip <head> and noise blocks that node-html-markdown won't remove
+  let html = input.replace(/<head[\s\S]*?<\/head>/gi, "");
+  let previousHtml = "";
+  while (previousHtml !== html) {
+    previousHtml = html;
+    html = html.replace(/<(script|style|nav|header|footer|noscript)[\s\S]*?<\/\1>/gi, "");
+  }
+
+  // 3. Delegate to node-html-markdown for full semantic conversion
+  const converted = NodeHtmlMarkdown.translate(html).trim();
+  if (!converted) return input;
+
+  // 4. Prepend <title> as # heading only if body has no <h1> of its own
+  const hasBodyH1 = /<h1[^>]*>[\s\S]*?<\/h1>/i.test(html);
+  const lines: string[] = [];
+  if (title && !hasBodyH1) lines.push(`# ${title}\n`);
+  lines.push(converted);
+  return lines.join("\n");
+}
+
 function jsonToMarkdown(json: string): string {
   let value: unknown;
   try {

package/extensions/llm-wiki/lib/source-packet.ts

@@ -2,7 +2,13 @@ import { mkdirSync, writeFileSync } from "node:fs";
 import { extname, join } from "node:path";
 import type { ExtensionAPI } from "@mariozechner/pi-coding-agent";
 import { appendEvent } from "./metadata.js";
-import { type ExtractedContent, extractUrlContent, fileExtractorFor } from "./source-extractors.js";
+import {
+  type ExtractedContent,
+  binaryExtractionFailureMessage,
+  detectBinaryMagicBytes,
+  extractUrlContent,
+  fileExtractorFor,
+} from "./source-extractors.js";
 import { type VaultPaths, exec, fmtDate, nextSourceId, readText, writeJson } from "./utils.js";
 
 /**
@@ -106,9 +112,28 @@ function fileCaptureSource(
     fallbackText: "",
     preserveOriginal: (packetPath) =>
       preserveFileOriginal(pi, packetPath, filePath, fileName, content, signal),
-    extract: async () => ({
-      extracted: await extractor.extract({ pi, filePath, content, signal }),
-    }),
+    extract: async () => {
+      // Guard: if we hit the generic catch-all extractor, check for binary magic bytes first
+      if (extractor.format === "file") {
+        const binaryFormat = await detectBinaryMagicBytes(filePath);
+        if (binaryFormat) {
+          return {
+            extracted: binaryExtractionFailureMessage(binaryFormat),
+            extractor: "magicBytes",
+            extraction_status: "unsupported" as const,
+          };
+        }
+      }
+
+      const extractedStr = await extractor.extract({ pi, filePath, content, signal });
+      const failed = extractedStr.includes("could not be converted");
+      return {
+        extracted: extractedStr,
+        extractor: extractor.extractorName ?? "passthrough",
+        extraction_status: (failed ? "failed" : "success") as "failed" | "success",
+        ...(extractor.content_type ? { content_type: extractor.content_type } : {}),
+      };
+    },
     manifest: () => ({
       title: fileName,
       file_path: filePath,
@@ -152,6 +177,9 @@ function finalizeCapture(
     captured: fmtDate(),
     packet_version: "1.0",
     ...source.manifest({ ...content, extracted }),
+    extractor: content.extractor ?? "passthrough",
+    extraction_status: content.extraction_status ?? "success",
+    ...(content.content_type ? { content_type: content.content_type } : {}),
   };
 
   writeFileSync(join(packet.packetPath, "extracted.md"), extracted, "utf-8");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zosmaai/pi-llm-wiki",
-  "version": "0.8.0",
+  "version": "0.8.2",
   "description": "Self-maintaining LLM Wiki for Pi — Karpathy-pattern knowledge base with immutable source capture, automated ingestion, search, linting, and Obsidian-compatible vault. auto-updating personal & company wiki.",
   "keywords": [
     "pi",
@@ -76,7 +76,8 @@
     "node": ">=18"
   },
   "dependencies": {
-    "@modelcontextprotocol/server": "^2.0.0-alpha.2"
+    "@modelcontextprotocol/server": "^2.0.0-alpha.2",
+    "node-html-markdown": "^2.0.0"
   },
   "devDependencies": {
     "@biomejs/biome": "^1.9.4",