wicked-brain 0.1.0

package/skills/wicked-brain-ingest/SKILL.md

@@ -0,0 +1,331 @@
+---
+name: wicked-brain:ingest
+description: |
+  Ingest source files into the brain as structured chunks. Handles text files
+  (md, txt, csv, html) deterministically and binary files (pdf, docx, pptx,
+  xlsx, images) via LLM vision. Dispatches a subagent for the heavy lifting.
+  
+  Use when: "ingest this file", "add to brain", "learn from this document",
+  "index this file", "brain ingest", "ingest this directory".
+---
+
+# wicked-brain:ingest
+
+You ingest source files into the brain by dispatching an ingest subagent.
+
+## Cross-Platform Notes
+
+Commands in this skill work on macOS, Linux, and Windows. When a command has
+platform differences, alternatives are shown. Your native tools (Read, Write,
+Grep, Glob) work everywhere — prefer them over shell commands when possible.
+
+For the brain path default:
+- macOS/Linux: ~/.wicked-brain
+- Windows: %USERPROFILE%\.wicked-brain
+
+## Config
+
+Read `_meta/config.json` for brain path and server port.
+If it doesn't exist, trigger wicked-brain:init.
+
+## Parameters
+
+- **source** (required): path to a file or directory to ingest
+
+## Process
+
+### Step 1: Assess scope
+
+Determine if the source is a single file or a directory.
+
+- **Single file**: dispatch a subagent to ingest it directly (Step 3a)
+- **Directory or multiple files**: use the batch script pattern (Step 3b)
+
+### Step 2: Copy source to raw/ (if not already there)
+
+If the source is outside the brain directory, symlink or copy it into `{brain_path}/raw/`.
+Prefer symlinks to avoid duplicating large files:
+- macOS/Linux: `ln -s "{source}" "{brain_path}/raw/{name}"`
+- Windows: `New-Item -ItemType SymbolicLink -Path "{brain_path}\raw\{name}" -Target "{source}"`
+
+### Step 3a: Single file ingest (subagent)
+
+Dispatch an ingest subagent with these instructions:
+
+```
+You are an ingest agent for the digital brain at {brain_path}.
+
+Source file: {source_path}
+Source name: {safe_name} (lowercase, hyphens for special chars)
+Server: http://localhost:{port}/api
+
+## Detect file type
+
+Check the file extension:
+- Text files (md, txt, csv, html, json): use deterministic extraction
+- Binary files (pdf, docx, pptx, xlsx, png, jpg, jpeg, gif, webp): use vision extraction
+
+## For TEXT files:
+
+1. Read the file content
+2. Split into chunks:
+   - Markdown: split on H1/H2 headings. If a section > 800 words, split further at paragraph breaks.
+   - Code files (.py, .js, .jsx, .ts): one chunk per file. Tag with language.
+   - Other text: split into paragraph groups of ~800 words.
+3. For each chunk, write to `{brain_path}/chunks/extracted/{safe_name}/chunk-NNN.md`
+
+## For BINARY files:
+
+1. Read the file (the LLM receives it natively as an attachment)
+2. Extract content by examining the document visually
+3. For PDFs: one chunk per logical section or every 3-5 pages
+4. For PPTX: one chunk per slide or slide group
+5. For DOCX: one chunk per section heading
+6. For XLSX: one chunk per sheet, render data as markdown tables
+7. For images: one chunk describing the visual content
+
+## Chunk format
+
+Each chunk file must have this structure:
+
+---
+source: {safe_name}
+source_type: {extension}
+chunk_id: {safe_name}/chunk-{NNN}
+content_type:
+  - text
+contains:
+  - {topic tags extracted from content}
+entities:
+  systems: [{named systems/platforms}]
+  people: [{people/roles}]
+  programs: [{programs/initiatives}]
+  metrics: ["{metric}: {value}"]
+confidence: {0.7 for text, 0.85 for vision}
+indexed_at: {current ISO timestamp}
+narrative_theme: {the "so what" in 8 words or fewer}
+---
+
+{Extracted content in markdown format}
+
+## After writing chunks, index them in the server:
+
+curl -s -X POST http://localhost:{port}/api \
+  -H "Content-Type: application/json" \
+  -d '{"action":"index","params":{"id":"{chunk_path}","path":"{chunk_path}","content":"{chunk_content}","brain_id":"{brain_id}"}}'
+
+## Report back
+
+State how many chunks were created and from what file.
+```
+
+### Step 3b: Batch ingest (script generation)
+
+When ingesting a directory or many files, **do not ingest files one-by-one in conversation**.
+Instead, write a batch script and run it. This preserves context and is dramatically faster.
+
+**The pattern:**
+
+1. Detect what runtime is available. Check in order:
+   - Node.js: `node --version`
+   - Python: `python3 --version` or `python --version`
+   - Shell: always available as fallback
+
+2. Write a script to the brain's `_meta/` directory that:
+   - Walks the source directory
+   - Filters by file extension (text types for deterministic, binary types for listing)
+   - For each text file: reads content, splits into chunks, writes chunk .md files, curls the index API
+   - For binary files: lists them for separate vision-based ingest
+   - Logs progress to stdout
+   - Writes a summary at the end
+
+3. Run the script:
+   ```bash
+   node {brain_path}/_meta/batch-ingest.mjs   # or .py or .sh
+   ```
+
+4. Read the output and report results to the user
+
+5. For any binary files identified, either:
+   - Dispatch individual vision ingest subagents for the most important ones
+   - Report the list and let the user choose which to vision-ingest
+
+**Example Node.js batch script structure:**
+
+```javascript
+#!/usr/bin/env node
+import { readFileSync, writeFileSync, mkdirSync, readdirSync, statSync, existsSync } from "node:fs";
+import { join, extname, basename, relative } from "node:path";
+import { createHash } from "node:crypto";
+
+const BRAIN = "{brain_path}";
+const PORT = {port};
+const BRAIN_ID = "{brain_id}";
+const SOURCE_DIR = "{source_dir}";
+
+const TEXT_EXT = new Set([".md",".txt",".csv",".html",".htm",".json",".py",".js",".jsx",".ts",".tsx",".sh"]);
+const BINARY_EXT = new Set([".pdf",".docx",".pptx",".xlsx",".png",".jpg",".jpeg",".gif",".webp"]);
+
+const binaryFiles = [];
+let totalChunks = 0;
+let totalFiles = 0;
+
+function safeName(name) {
+  return name.toLowerCase().replace(/[^a-z0-9.-]/g, "-").replace(/-+/g, "-").replace(/^-|-$/g, "");
+}
+
+function hash(content) {
+  return createHash("sha256").update(content).digest("hex").slice(0, 16);
+}
+
+async function indexChunk(id, path, content) {
+  try {
+    await fetch(`http://localhost:${PORT}/api`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ action: "index", params: { id, path, content, brain_id: BRAIN_ID } }),
+    });
+  } catch (e) {
+    console.error(`  Failed to index ${id}: ${e.message}`);
+  }
+}
+
+function splitMarkdown(text) {
+  const sections = text.split(/(?=^#{1,2}\s)/m).filter(s => s.trim());
+  if (sections.length === 0) return [text];
+  // Sub-split sections > 800 words
+  const result = [];
+  for (const section of sections) {
+    const words = section.split(/\s+/).length;
+    if (words > 800) {
+      const paragraphs = section.split(/\n\n+/);
+      let current = [];
+      let count = 0;
+      for (const p of paragraphs) {
+        const w = p.split(/\s+/).length;
+        if (count + w > 800 && current.length > 0) {
+          result.push(current.join("\n\n"));
+          current = [p];
+          count = w;
+        } else {
+          current.push(p);
+          count += w;
+        }
+      }
+      if (current.length > 0) result.push(current.join("\n\n"));
+    } else {
+      result.push(section);
+    }
+  }
+  return result;
+}
+
+function splitText(text) {
+  const paragraphs = text.split(/\n\n+/);
+  const chunks = [];
+  let current = [];
+  let count = 0;
+  for (const p of paragraphs) {
+    const w = p.split(/\s+/).length;
+    if (count + w > 800 && current.length > 0) {
+      chunks.push(current.join("\n\n"));
+      current = [p];
+      count = w;
+    } else {
+      current.push(p);
+      count += w;
+    }
+  }
+  if (current.length > 0) chunks.push(current.join("\n\n"));
+  return chunks.length > 0 ? chunks : [text];
+}
+
+async function ingestFile(filePath) {
+  const ext = extname(filePath).toLowerCase();
+  const rel = relative(SOURCE_DIR, filePath);
+  const name = safeName(rel);
+  const chunkDir = join(BRAIN, "chunks", "extracted", name);
+  mkdirSync(chunkDir, { recursive: true });
+
+  const content = readFileSync(filePath, "utf-8");
+  const chunks = ext === ".md" ? splitMarkdown(content) : splitText(content);
+
+  for (let i = 0; i < chunks.length; i++) {
+    const chunkId = `${name}/chunk-${String(i + 1).padStart(3, "0")}`;
+    const chunkPath = `chunks/extracted/${chunkId}.md`;
+    const ts = new Date().toISOString();
+    const keywords = [...new Set(chunks[i].toLowerCase().replace(/[^a-z0-9\s-]/g,"").split(/\s+/).filter(w => w.length > 5))].slice(0, 10);
+
+    const frontmatter = [
+      "---",
+      `source: ${basename(filePath)}`,
+      `source_type: ${ext.slice(1)}`,
+      `chunk_id: ${chunkId}`,
+      "content_type:",
+      "  - text",
+      "contains:",
+      ...keywords.map(k => `  - ${k}`),
+      `confidence: 0.7`,
+      `indexed_at: "${ts}"`,
+      "---",
+    ].join("\n");
+
+    const fullContent = `${frontmatter}\n\n${chunks[i]}`;
+    writeFileSync(join(BRAIN, chunkPath), fullContent);
+    await indexChunk(chunkPath, chunkPath, chunks[i]);
+    totalChunks++;
+  }
+
+  totalFiles++;
+  console.log(`  ${name}: ${chunks.length} chunks`);
+}
+
+function walk(dir, callback) {
+  for (const entry of readdirSync(dir, { withFileTypes: true })) {
+    const full = join(dir, entry.name);
+    if (entry.name.startsWith(".") || entry.name === "node_modules" || entry.name === "__pycache__" || entry.name === "package-lock.json") continue;
+    if (entry.isDirectory()) walk(full, callback);
+    else if (entry.isFile()) callback(full);
+  }
+}
+
+console.log(`Ingesting from ${SOURCE_DIR}...`);
+
+walk(SOURCE_DIR, (filePath) => {
+  const ext = extname(filePath).toLowerCase();
+  if (TEXT_EXT.has(ext)) {
+    try { ingestFile(filePath); } catch (e) { console.error(`  Error: ${filePath}: ${e.message}`); }
+  } else if (BINARY_EXT.has(ext)) {
+    binaryFiles.push(filePath);
+  }
+});
+
+// Wait for all index operations
+await new Promise(r => setTimeout(r, 1000));
+
+console.log(`\nDone: ${totalFiles} files, ${totalChunks} chunks indexed`);
+if (binaryFiles.length > 0) {
+  console.log(`\nBinary files needing vision ingest (${binaryFiles.length}):`);
+  for (const f of binaryFiles) console.log(`  ${f}`);
+}
+```
+
+This pattern should be used **whenever more than 5 files need processing**. It:
+- Preserves agent context (no 50 Read/Write/Bash cycles)
+- Runs fast (single process, no round-trips)
+- Works cross-platform (Node.js is available everywhere wicked-brain runs)
+- Reports results the agent can summarize
+
+### Step 4: Archive on re-ingest
+
+If previous chunks exist for a source, archive them first.
+Use the agent's native move/rename capability, or shell equivalents:
+- macOS/Linux: `mv "{brain_path}/chunks/extracted/{safe_name}" "{brain_path}/chunks/extracted/{safe_name}.archived-$(date +%s)"`
+- Windows: `Rename-Item "{brain_path}\chunks\extracted\{safe_name}" "{safe_name}.archived-{timestamp}"`
+
+### Step 5: Report to user
+
+After the subagent or batch script completes, summarize:
+- "{N} text files ingested, {M} chunks created"
+- "{K} binary files identified for vision ingest" (if any)
+- Offer to vision-ingest the most important binary files

package/skills/wicked-brain-init/SKILL.md

@@ -0,0 +1,110 @@
+---
+name: wicked-brain:init
+description: |
+  Initialize a new digital brain. Creates the directory structure, brain.json,
+  and config. Auto-triggered on first use of any brain skill when no config exists.
+  
+  Use when: "set up a brain", "create a brain", "brain init", or when any brain
+  skill detects no config.
+---
+
+# wicked-brain:init
+
+You initialize a new digital brain on the filesystem.
+
+## Cross-Platform Notes
+
+Commands in this skill work on macOS, Linux, and Windows. When a command has
+platform differences, alternatives are shown. Your native tools (Read, Write,
+Grep, Glob) work everywhere — prefer them over shell commands when possible.
+
+For the brain path default:
+- macOS/Linux: ~/.wicked-brain
+- Windows: %USERPROFILE%\.wicked-brain
+
+## When to use
+
+- User explicitly asks to create/initialize a brain
+- Another brain skill detected no `_meta/config.json` and redirected here
+
+## Process
+
+### Step 1: Ask the user
+
+Ask these questions (provide defaults):
+
+1. "Where should your brain live?"
+   - Default (macOS/Linux): `~/.wicked-brain`
+   - Default (Windows): `%USERPROFILE%\.wicked-brain`
+2. "What should this brain be called?" — Default: directory name
+
+### Step 2: Create directory structure
+
+Use your native Write/mkdir tools to create these directories and files.
+
+Directories to create (create each with its parent directories):
+- `{brain_path}/raw`
+- `{brain_path}/chunks/extracted`
+- `{brain_path}/chunks/inferred`
+- `{brain_path}/wiki/concepts`
+- `{brain_path}/wiki/topics`
+- `{brain_path}/_meta`
+
+Shell equivalents if needed:
+```bash
+# macOS/Linux
+mkdir -p {brain_path}/raw {brain_path}/chunks/extracted {brain_path}/chunks/inferred \
+  {brain_path}/wiki/concepts {brain_path}/wiki/topics {brain_path}/_meta
+```
+```powershell
+# Windows PowerShell
+New-Item -ItemType Directory -Force -Path "{brain_path}\raw","{brain_path}\chunks\extracted","{brain_path}\chunks\inferred","{brain_path}\wiki\concepts","{brain_path}\wiki\topics","{brain_path}\_meta"
+```
+
+### Step 3: Write brain.json
+
+Write to `{brain_path}/brain.json`:
+```json
+{
+  "schema": 1,
+  "id": "{id}",
+  "name": "{name}",
+  "parents": [],
+  "links": []
+}
+```
+
+Where `{id}` is the directory name (lowercase, hyphens for spaces) and `{name}` is what the user provided.
+
+### Step 4: Write config
+
+Write to `{brain_path}/_meta/config.json`:
+```json
+{
+  "brain_path": "{absolute_path}",
+  "server_port": 4242,
+  "installed_clis": []
+}
+```
+
+### Step 5: Initialize the event log
+
+Use your Write tool to create an empty file at `{brain_path}/_meta/log.jsonl`.
+
+Shell equivalents if needed:
+```bash
+# macOS/Linux
+touch {brain_path}/_meta/log.jsonl
+```
+```powershell
+# Windows PowerShell
+New-Item -ItemType File -Force -Path "{brain_path}\_meta\log.jsonl"
+```
+
+### Step 6: Confirm
+
+Tell the user:
+"Brain initialized at `{brain_path}`. You can now:
+- `wicked-brain:ingest` to add source files
+- `wicked-brain:search` to search content
+- `wicked-brain:status` to check brain health"

package/skills/wicked-brain-lint/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: wicked-brain:lint
+description: |
+  Check brain health and fix issues. Dispatches a lint subagent that runs
+  deterministic checks (broken links, orphans, stale entries) and semantic
+  analysis (inconsistencies, gaps, tag misalignment).
+  
+  Use when: "lint the brain", "check brain health", "brain lint",
+  "find issues in the brain".
+---
+
+# wicked-brain:lint
+
+You check brain quality by dispatching a lint subagent.
+
+## Cross-Platform Notes
+
+Commands in this skill work on macOS, Linux, and Windows. When a command has
+platform differences, alternatives are shown. Your native tools (Read, Write,
+Grep, Glob) work everywhere — prefer them over shell commands when possible.
+
+For the brain path default:
+- macOS/Linux: ~/.wicked-brain
+- Windows: %USERPROFILE%\.wicked-brain
+
+## Config
+
+Read `_meta/config.json` for brain path and server port.
+If it doesn't exist, trigger wicked-brain:init.
+
+## Process
+
+Dispatch a lint subagent with these instructions:
+
+```
+You are a quality assurance agent for the digital brain at {brain_path}.
+Server: http://localhost:{port}/api
+
+## Pass 1: Deterministic checks
+
+### Broken wikilinks
+Find all [[wikilinks]] in wiki and chunk files, check if targets exist.
+Use your Grep tool (preferred):
+- Pattern: `\[\[[^\]]*\]\]`
+- Search in: `{brain_path}/wiki/` and `{brain_path}/chunks/`
+
+Shell fallback:
+- macOS/Linux: `grep -roh '\[\[[^]]*\]\]' {brain_path}/wiki/ {brain_path}/chunks/ 2>/dev/null | sort -u`
+- Windows: `findstr /s /r "\[\[" "{brain_path}\wiki\*.md" "{brain_path}\chunks\*.md" 2>nul`
+
+For each link, use the Read tool to check if the target file exists.
+
+### Orphan chunks
+Use your Glob tool to find all chunk files in `{brain_path}/chunks/**/*.md`.
+Then use your Grep tool to check which chunk IDs appear in wiki files.
+
+Shell fallback:
+- macOS/Linux:
+  ```bash
+  find {brain_path}/chunks -name "chunk-*.md" -type f
+  grep -rl "chunk-" {brain_path}/wiki/ 2>/dev/null
+  ```
+- Windows:
+  ```powershell
+  Get-ChildItem -Recurse -Filter "chunk-*.md" "{brain_path}\chunks"
+  findstr /s /m "chunk-" "{brain_path}\wiki\*.md" 2>nul
+  ```
+
+### Stale entries
+Compare source file modification times with chunk creation times.
+
+### Missing frontmatter
+Check each chunk has required frontmatter fields (source, chunk_id, confidence, indexed_at).
+
+## Pass 2: Semantic analysis
+
+Read a sample of chunks and wiki articles. Check:
+- Are tags consistent? (same concept tagged differently in different chunks)
+- Are there factual contradictions between articles?
+- Are there implicit connections that should be explicit [[links]]?
+- What topics have chunks but no wiki article? (coverage gaps)
+
+## Report
+
+For each issue found:
+- **severity**: error | warning | info
+- **type**: broken_link | orphan | stale | missing_field | inconsistency | gap
+- **path**: which file
+- **message**: what's wrong
+- **fix**: suggested fix (or "auto-fixed" if you fixed it)
+
+Auto-fix broken links and missing fields where possible. Report everything else.
+```

package/skills/wicked-brain-query/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: wicked-brain:query
+description: |
+  Answer questions by searching and synthesizing brain content. Dispatches a
+  query subagent that searches, reads, follows links, and produces a cited answer.
+  
+  Use when: user asks a question that could be answered from brain content,
+  "ask the brain", "brain query", "what does my brain say about".
+---
+
+# wicked-brain:query
+
+You answer questions from the brain's content by dispatching a query subagent.
+
+## Config
+
+Read `_meta/config.json` for brain path and server port.
+If it doesn't exist, trigger wicked-brain:init.
+
+## Parameters
+
+- **question** (required): the question to answer
+
+## Process
+
+Dispatch a query subagent with these instructions:
+
+```
+You are a research agent for the digital brain at {brain_path}.
+Server: http://localhost:{port}/api
+
+Question: "{question}"
+
+## Step 1: Search
+
+Search the brain for relevant content:
+```bash
+curl -s -X POST http://localhost:{port}/api \
+  -H "Content-Type: application/json" \
+  -d '{"action":"search","params":{"query":"{question}","limit":10}}'
+```
+
+Also search with grep for exact phrases:
+```bash
+grep -rl "{key_terms}" {brain_path}/chunks/ {brain_path}/wiki/ 2>/dev/null | head -10
+```
+
+## Step 2: Progressive read
+
+Read the top 3-5 results at depth 1 first (just frontmatter + summary).
+Then read the most promising 1-3 at depth 2 (full content).
+
+Use the Read tool for each file. Parse frontmatter between `---` lines.
+
+## Step 3: Follow links
+
+Check the content for [[wikilinks]]. If following them would provide useful context:
+- For local links [[path]]: read that file
+- For cross-brain links [[brain::path]]: check if that brain is accessible
+
+Check backlinks — what else references the content you found:
+```bash
+curl -s -X POST http://localhost:{port}/api \
+  -H "Content-Type: application/json" \
+  -d '{"action":"backlinks","params":{"id":"{result_path}"}}'
+```
+
+## Step 4: Synthesize answer
+
+Combine what you found into a clear answer. Requirements:
+- Cite sources: [source: {path}] for every factual claim
+- If evidence is insufficient, say so explicitly
+- If sources conflict, note the contradiction
+- Keep the answer concise — the user asked a question, not for a report
+
+## Report format
+
+Answer the question directly, then list sources:
+
+"{Answer text with [source: path] citations}"
+
+Sources:
+- {path}: {one-line description of what it contributed}
+- {path}: {one-line description}
+```

package/skills/wicked-brain-read/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: wicked-brain:read
+description: |
+  Read a chunk or wiki article from the brain with progressive loading.
+  Depth 0: frontmatter + stats. Depth 1: + summary + headings. Depth 2: full content.
+  
+  Use when: "read this chunk", "show me the article", "brain read", following up
+  from search results, or when needing to inspect brain content.
+---
+
+# wicked-brain:read
+
+You read content from the digital brain with progressive loading. Never return
+more than the user or calling skill needs.
+
+## Config
+
+Read the brain path from `_meta/config.json`. If it doesn't exist, trigger wicked-brain:init.
+
+## Parameters
+
+- **path** (required): relative path within the brain (e.g., `wiki/concepts/kg.md`)
+- **depth** (default: 1): how much to return
+  - 0: frontmatter + word count + link count (~5 tokens)
+  - 1: frontmatter + first paragraph + section headings (~50-100 tokens)
+  - 2: full content (variable)
+- **sections** (optional, depth 2 only): list of section headings to extract (e.g., `## Methods`)
+
+## Process
+
+### Step 1: Read the file
+
+Use the Read tool to read `{brain_path}/{path}`.
+
+### Step 2: Parse frontmatter
+
+The file starts with YAML between `---` delimiters:
+```
+---
+key: value
+---
+Body content here...
+```
+
+Split the file on the second `---` line. Everything before is frontmatter, everything after is body.
+
+### Step 3: Count stats
+
+- **word_count**: split body on whitespace, count words
+- **link_count**: count occurrences of `[[` in the body (each `[[...]]` is one link)
+- **related**: extract all `[[target]]` and `[[brain::target]]` patterns
+
+### Step 4: Return at requested depth
+
+**Depth 0:**
+Report only:
+- Frontmatter fields
+- Word count: {N}
+- Links: {N}
+- Related: [list of link targets]
+
+**Depth 1:**
+Report depth 0 plus:
+- **Summary**: the first non-empty paragraph after frontmatter (skip headings)
+- **Sections**: list all lines starting with `#`, `##`, `###`
+
+**Depth 2:**
+Report the full body content. If `sections` parameter is provided, extract only
+the requested sections (from heading to next heading of same or higher level).
+
+## Always include deeper hints
+
+If returning at depth 0 or 1, suggest: "Use wicked-brain:read at depth {next} for more detail."