xtrm-tools 0.5.0

package/skills/sync-docs/references/doc-structure.md

@@ -0,0 +1,99 @@
+# docs/ Structure Guide
+
+This reference defines what belongs in each focused docs/ file vs README.md.
+
+## The Rule
+
+**README.md** is an entry point — a quick-start and orientation map. It should be < 200 lines. Anything that requires more than a few bullet points belongs in a focused docs/ file.
+
+**docs/** holds focused, public-facing SSOT files for each subsystem. Each file:
+- Has YAML frontmatter (see `schema.md`)
+- Covers exactly one subsystem or concern
+- Is linked from README.md with a single line
+
+---
+
+## What Goes Where
+
+| Content Type | Location | When to Create |
+|---|---|---|
+| Quick start, one-liner install | `README.md` | Always |
+| Feature overview table | `README.md` | Always, < 20 lines |
+| Hook events, scripts, behavior | `docs/hooks.md` | When hooks/ dir exists |
+| Pi/Copilot extension catalog | `docs/pi-extensions.md` | When config/pi/extensions/ exists |
+| System architecture, components | `docs/architecture.md` | When > 2 major subsystems |
+| Policy rules and enforcement | `docs/policies.md` | When policies/ dir exists |
+| MCP server config and usage | `docs/mcp-servers.md` | When .mcp.json or mcp servers exist |
+| Skills catalog | `docs/skills.md` | When skills/ dir has > 5 skills |
+| CLI commands reference | `docs/cli-reference.md` | When CLI has > 10 commands |
+| Troubleshooting and FAQs | `docs/troubleshooting.md` | When recurring issues exist |
+| In-progress work plans | `docs/plans/` | Always for planning docs |
+
+---
+
+## Standard File Shapes
+
+### docs/hooks.md
+Covers: hook events, what triggers each hook, which scripts run, output format.
+Section outline:
+```
+## Overview
+## Hook Events
+## Scripts Reference
+## Adding a New Hook
+```
+
+### docs/pi-extensions.md
+Covers: what Pi extensions are installed, their events, behavior, and configuration.
+Section outline:
+```
+## Overview
+## Installed Extensions
+## Extension Events
+## Configuration
+```
+
+### docs/architecture.md
+Covers: system overview, key components, data flow, dependency diagram.
+Section outline:
+```
+## System Overview
+## Key Components
+## Data Flow
+## Directory Structure
+```
+
+### docs/policies.md
+Covers: what policies exist, what each enforces, runtime targets.
+Section outline:
+```
+## Overview
+## Policy Files
+## Policy Compiler
+## Adding a Policy
+```
+
+---
+
+## Detecting When README is Too Big
+
+The `doc_structure_analyzer.py` script flags README as `BLOATED` when:
+1. Line count > 200, AND
+2. At least one section has a matching `docs/` file that doesn't exist yet
+
+The `EXTRACTABLE` status means sections are present but README isn't over threshold yet — worth noting but not urgent.
+
+---
+
+## Extraction Process
+
+When extracting a section from README to docs/:
+
+1. Identify the target section heading and its body in `README.md`
+2. Create `docs/X.md` with `validate_doc.py --generate`
+3. Move the section content into the new file
+4. Replace the original README section with:
+   ```markdown
+   See [docs/X.md](docs/X.md) for the full reference.
+   ```
+5. Run `validate_doc.py docs/X.md` to confirm schema

package/skills/sync-docs/references/schema.md

@@ -0,0 +1,103 @@
+# docs/ File Schema
+
+Every file in `docs/` must have YAML frontmatter. This is enforced by `validate_doc.py`.
+
+## Frontmatter Fields
+
+```yaml
+---
+title: Human Readable Title          # required — clear, descriptive
+scope: kebab-case-identifier         # required — unique slug for this doc
+category: reference                  # required — see valid values below
+version: 1.0.0                       # required — semver (x.y.z)
+updated: 2026-03-18                  # required — ISO date of last update
+description: "One-line summary"      # optional
+source_of_truth_for:                 # optional — glob patterns this doc describes
+  - "hooks/**/*.mjs"
+  - "policies/*.json"
+domain: [hooks, claude, policy]      # optional — cross-cutting tags
+---
+```
+
+## Required Fields
+
+| Field | Type | Description | Example |
+|-------|------|-------------|---------|
+| `title` | string | Clear, descriptive title | "Hooks Reference" |
+| `scope` | string | Unique kebab-case slug | "hooks" |
+| `category` | enum | Document type | "reference" |
+| `version` | semver | x.y.z — bump on every update | "1.2.0" |
+| `updated` | date | YYYY-MM-DD of last change | "2026-03-18" |
+
+## Category Values
+
+| Value | Purpose |
+|---|---|
+| `reference` | Look-up table, cheat sheet, technical spec |
+| `guide` | Step-by-step how-to documentation |
+| `architecture` | System design and component overview |
+| `api` | API contracts, interfaces, schemas |
+| `plan` | Implementation plan or roadmap |
+| `overview` | Summary introduction to a subsystem |
+
+## Optional Fields
+
+### source_of_truth_for
+A list of glob patterns (fnmatch syntax) pointing to the source files this doc describes. Used by `drift_detector.py` and `doc_structure_analyzer.py` to detect when the source changed but the doc wasn't updated.
+
+```yaml
+source_of_truth_for:
+  - "hooks/**/*.mjs"
+  - "policies/*.json"
+  - "config/pi/extensions/**/*.ts"
+```
+
+### domain
+Tags for cross-cutting concerns. Useful for searching related docs.
+
+```yaml
+domain: [hooks, claude, pi, enforcement]
+```
+
+## Version Bumping Rules
+
+| Change | Bump |
+|---|---|
+| Fix a typo or outdated detail | `patch` (1.0.0 → 1.0.1) |
+| Add a new section | `minor` (1.0.0 → 1.1.0) |
+| Full rewrite or restructure | `major` (1.0.0 → 2.0.0) |
+
+Always update `updated:` to today's date alongside the version bump.
+
+## INDEX Block
+
+`validate_doc.py` auto-generates a navigation table from `##` headings:
+
+```markdown
+<!-- INDEX: auto-generated by validate_doc.py — do not edit manually -->
+| Section | Summary |
+|---|---|
+| [Overview](#overview) | ... |
+| [Hook Events](#hook-events) | ... |
+<!-- END INDEX -->
+```
+
+This block is regenerated every time `validate_doc.py` runs on the file. Do not edit it manually.
+
+## Complete Example
+
+```yaml
+---
+title: Hooks Reference
+scope: hooks
+category: reference
+version: 1.1.0
+updated: 2026-03-18
+description: "All hook events, scripts, and behavior for the xtrm plugin"
+source_of_truth_for:
+  - "hooks/**/*.mjs"
+  - "policies/beads.json"
+  - "policies/main-guard.json"
+domain: [hooks, claude, enforcement]
+---
+```

package/skills/sync-docs/scripts/changelog/add_entry.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python3
+"""
+Add entry to CHANGELOG.md [Unreleased] section.
+
+Automatically:
+- Places entry in correct category
+- Creates category if missing
+- Maintains category ordering per Keep a Changelog
+- Preserves existing entries
+"""
+
+import re
+import sys
+from enum import Enum
+from pathlib import Path
+from typing import Optional
+
+
+class ChangeCategory(Enum):
+    """Keep a Changelog categories in proper order."""
+    ADDED = "Added"
+    CHANGED = "Changed"
+    DEPRECATED = "Deprecated"
+    REMOVED = "Removed"
+    FIXED = "Fixed"
+    SECURITY = "Security"
+
+
+CATEGORY_ORDER = [cat.value for cat in ChangeCategory]
+
+
+def add_entry(changelog_content: str, category: ChangeCategory, description: str) -> str:
+    """
+    Add entry to [Unreleased] section under specified category.
+
+    Args:
+        changelog_content: Full CHANGELOG.md content
+        category: ChangeCategory enum value
+        description: Entry text (without leading "- ")
+
+    Returns:
+        Updated changelog content
+    """
+    lines = changelog_content.splitlines()
+
+    # Find [Unreleased] section
+    unreleased_idx = None
+    for i, line in enumerate(lines):
+        if re.match(r"^## \[Unreleased\]", line):
+            unreleased_idx = i
+            break
+
+    if unreleased_idx is None:
+        raise ValueError("CHANGELOG missing [Unreleased] section")
+
+    # Find next version section (end of Unreleased)
+    next_version_idx = len(lines)
+    for i in range(unreleased_idx + 1, len(lines)):
+        if re.match(r"^## \[.+\]", lines[i]):
+            next_version_idx = i
+            break
+
+    # Find or create category section
+    category_name = category.value
+    category_idx = None
+
+    for i in range(unreleased_idx + 1, next_version_idx):
+        if lines[i].strip() == f"### {category_name}":
+            category_idx = i
+            break
+
+    if category_idx is None:
+        # Category doesn't exist, create it in proper order
+        category_idx = _insert_category_in_order(
+            lines,
+            unreleased_idx,
+            next_version_idx,
+            category_name
+        )
+        # Update next_version_idx as lines shifted
+        # Wait, if I insert, I need to know how many lines. 
+        # _insert_category_in_order returns the index of the new category header.
+        # But if it inserted blank lines, the indices after it shifted. 
+        # But next_version_idx is an index, so if I insert BEFORE it, it shifts.
+        # Let's check _insert_category_in_order implementation logic.
+        # It mutates 'lines' list.
+        # If it inserts 2 lines, next_version_idx should increase by 2.
+        # But wait, next_version_idx was calculated based on initial list.
+        # I should probably re-calculate or just use the return value.
+        # But wait, the function mutates the list.
+        # Let's look at the implementation below.
+        
+        # Recalculate next_version_idx to be safe? 
+        # Or rely on finding the category index.
+        pass
+
+    # Re-find next version idx just to be safe if lines shifted
+    for i in range(unreleased_idx + 1, len(lines)):
+        if re.match(r"^## \[.+\]", lines[i]):
+            next_version_idx = i
+            break
+    else:
+        next_version_idx = len(lines)
+        
+    # Re-find category index
+    for i in range(unreleased_idx + 1, next_version_idx):
+        if lines[i].strip() == f"### {category_name}":
+            category_idx = i
+            break
+
+    # Find where to insert the entry (after category header)
+    insert_idx = category_idx + 1
+
+    # Skip existing entries to add at end of category
+    while insert_idx < len(lines) and insert_idx < next_version_idx and lines[insert_idx].strip().startswith("- "):
+        insert_idx += 1
+
+    # Insert the new entry
+    entry_line = f"- {description}"
+    lines.insert(insert_idx, entry_line)
+
+    return '\n'.join(lines)
+
+
+def _insert_category_in_order(
+    lines: list,
+    unreleased_idx: int,
+    next_version_idx: int,
+    category_name: str
+) -> int:
+    """
+    Insert category header in proper Keep a Changelog order.
+
+    Returns:
+        Index where category header was inserted
+    """
+    category_order_idx = CATEGORY_ORDER.index(category_name)
+
+    # Find existing categories in Unreleased section
+    existing_categories = []
+    # Note: next_version_idx is based on current lines state
+    for i in range(unreleased_idx + 1, next_version_idx):
+        match = re.match(r"^### (.+)$", lines[i])
+        if match:
+            cat = match.group(1)
+            if cat in CATEGORY_ORDER:
+                existing_categories.append((i, cat))
+
+    # Find insertion point
+    insert_idx = next_version_idx
+    for idx, cat in existing_categories:
+        cat_order_idx = CATEGORY_ORDER.index(cat)
+        if category_order_idx < cat_order_idx:
+            # Insert before this category
+            insert_idx = idx
+            break
+
+    # Insert category header with blank line before if not first
+    if existing_categories:
+        lines.insert(insert_idx, "")
+        insert_idx += 1
+    else:
+        # First category, add blank line after [Unreleased]
+        lines.insert(unreleased_idx + 1, "")
+        insert_idx = unreleased_idx + 2
+
+    lines.insert(insert_idx, f"### {category_name}")
+
+    return insert_idx
+
+
+def add_entry_to_file(
+    filepath: Path,
+    category: ChangeCategory,
+    description: str
+) -> None:
+    """Add entry to CHANGELOG file."""
+    if not filepath.exists():
+        raise FileNotFoundError(f"CHANGELOG not found: {filepath}")
+
+    content = filepath.read_text(encoding='utf-8')
+    updated = add_entry(content, category, description)
+    filepath.write_text(updated, encoding='utf-8')
+
+    print(f"✅ Added to {filepath.name}:")
+    print(f"   [{category.value}] {description}")
+
+
+def main():
+    """CLI entry point."""
+    if len(sys.argv) != 4:
+        print("Usage: add_entry.py <changelog_file> <category> <description>")
+        print("")
+        print("Categories: Added, Changed, Deprecated, Removed, Fixed, Security")
+        print("")
+        print("Example:")
+        print('  add_entry.py CHANGELOG.md Added "New semantic search feature"')
+        sys.exit(1)
+
+    filepath = Path(sys.argv[1])
+    category_str = sys.argv[2]
+    description = sys.argv[3]
+
+    # Validate category
+    try:
+        category = ChangeCategory[category_str.upper()]
+    except KeyError:
+        print(f"ERROR: Invalid category '{category_str}'")
+        print(f"Valid: {', '.join(CATEGORY_ORDER)}")
+        sys.exit(1)
+
+    add_entry_to_file(filepath, category, description)
+
+
+if __name__ == "__main__":
+    main()

package/skills/sync-docs/scripts/context_gatherer.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""
+Gather project context for documentation sync.
+
+Collects:
+  - Recently closed bd issues (if .beads/ exists)
+  - Recently merged PRs (via git log)
+  - bd memories persisted this cycle (bd kv list)
+  - Stale docs/ files (via sync-docs drift_detector.py)
+
+Outputs JSON to stdout. Safe to run in any project — degrades gracefully
+when bd or drift detection tools are unavailable.
+
+Usage:
+  context_gatherer.py [--since=30]
+
+  --since=N   Look back N commits for git context (default: 30)
+"""
+
+import sys
+import json
+import subprocess
+import time
+from pathlib import Path
+from datetime import datetime, timezone
+
+
+def run(cmd: list, cwd: str | None = None, timeout: int = 8) -> str | None:
+    """Run a command, return stdout or None on failure."""
+    try:
+        result = subprocess.run(
+            cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout
+        )
+        if result.returncode == 0:
+            return result.stdout.strip()
+        return None
+    except Exception:
+        return None
+
+
+def find_project_root() -> Path:
+    """Walk up from cwd looking for .git."""
+    p = Path.cwd()
+    for parent in [p, *p.parents]:
+        if (parent / ".git").exists():
+            return parent
+    return p
+
+
+def find_main_repo_root(root: Path) -> Path:
+    """For git worktrees, resolve the main repo root from the .git file.
+
+    In a worktree, .git is a file: "gitdir: /path/to/main/.git/worktrees/<name>"
+    Navigate up two levels to reach the main .git, then one more for the repo root.
+    """
+    git_path = root / ".git"
+    if git_path.is_file():
+        content = git_path.read_text(encoding="utf-8").strip()
+        if content.startswith("gitdir:"):
+            worktree_git = Path(content[len("gitdir:"):].strip())
+            main_git = worktree_git.parent.parent
+            return main_git.parent
+    return root
+
+
+def ensure_dolt_server(cwd: str) -> bool:
+    """Ensure the Dolt server is running. Start it if not. Returns True if ready."""
+    test = run(["bd", "dolt", "test"], cwd=cwd, timeout=5)
+    if test is not None:
+        return True
+
+    try:
+        subprocess.run(
+            ["bd", "dolt", "start"],
+            cwd=cwd,
+            capture_output=True,
+            text=True,
+            timeout=15,
+        )
+    except Exception:
+        return False
+
+    for _ in range(6):
+        time.sleep(1)
+        if run(["bd", "dolt", "test"], cwd=cwd, timeout=3) is not None:
+            return True
+
+    return False
+
+
+def has_beads(root: Path) -> bool:
+    return (root / ".beads").exists()
+
+
+def gather_bd_closed(cwd: str) -> list[dict]:
+    """Get recently closed bd issues."""
+    out = run(["bd", "list", "--status=closed"], cwd=cwd)
+    if not out:
+        return []
+
+    issues = []
+    for line in out.splitlines():
+        line = line.strip()
+        if line.startswith("✓") or "closed" in line.lower():
+            parts = line.lstrip("✓ ").split()
+            if len(parts) >= 2:
+                issue_id = parts[0]
+                title_start = 2 if len(parts) > 2 and parts[1].startswith("P") else 1
+                title = " ".join(parts[title_start:])
+                issues.append({"id": issue_id, "title": title})
+
+    return issues[:20]
+
+
+def gather_bd_memories(cwd: str) -> list[dict]:
+    """Read bd memories via bd kv list, filtering memory.* keys."""
+    out = run(["bd", "kv", "list"], cwd=cwd)
+    if not out:
+        return []
+
+    memories = []
+    for line in out.splitlines():
+        stripped = line.strip()
+        if not stripped.startswith("memory."):
+            continue
+        if " = " in stripped:
+            key, _, value = stripped.partition(" = ")
+            memories.append({"key": key.strip(), "value": value.strip()})
+        else:
+            memories.append({"key": stripped, "value": ""})
+
+    return memories[:20]
+
+
+def gather_merged_prs(root: Path, since_n: int) -> list[dict]:
+    """Get merged PRs from git log."""
+    out = run(
+        ["git", "log", f"-{since_n}", "--merges", "--oneline", "--format=%H|%s|%ci"],
+        cwd=str(root),
+    )
+    if not out:
+        return []
+
+    prs = []
+    for line in out.splitlines():
+        parts = line.split("|", 2)
+        if len(parts) == 3:
+            sha, subject, date = parts
+            prs.append({"sha": sha[:8], "subject": subject.strip(), "date": date.strip()})
+    return prs[:10]
+
+
+def gather_recent_commits(root: Path, since_n: int) -> list[dict]:
+    """Get recent non-merge commits for context."""
+    out = run(
+        ["git", "log", f"-{since_n}", "--no-merges", "--oneline", "--format=%H|%s|%ci"],
+        cwd=str(root),
+    )
+    if not out:
+        return []
+
+    commits = []
+    for line in out.splitlines():
+        parts = line.split("|", 2)
+        if len(parts) == 3:
+            sha, subject, date = parts
+            commits.append({"sha": sha[:8], "subject": subject.strip(), "date": date.strip()})
+    return commits[:15]
+
+
+def gather_docs_drift(root: Path, since_n: int) -> dict:
+    """Run sync-docs drift_detector.py and capture stale docs report."""
+    candidates = [
+        Path.home() / ".claude/skills/sync-docs/scripts/drift_detector.py",
+        root / "skills/sync-docs/scripts/drift_detector.py",
+        Path(__file__).parent / "drift_detector.py",
+    ]
+    detector = next((p for p in candidates if p.exists()), None)
+    if not detector:
+        return {"available": False, "stale": []}
+
+    out = run([sys.executable, str(detector), "scan", "--since", str(since_n), "--json"], cwd=str(root))
+    if out is None:
+        return {"available": False, "stale": []}
+
+    try:
+        data = json.loads(out)
+        return {
+            "available": True,
+            "stale": data.get("stale", []),
+            "count": data.get("count", 0),
+            "raw": data,
+        }
+    except json.JSONDecodeError:
+        return {"available": True, "stale": [], "raw": out}
+
+
+def main() -> None:
+    since_n = 30
+    for arg in sys.argv[1:]:
+        if arg.startswith("--since="):
+            try:
+                since_n = int(arg.split("=", 1)[1])
+            except ValueError:
+                pass
+
+    root = find_project_root()
+    main_root = find_main_repo_root(root)
+    bd_cwd = str(main_root)
+    bd_available = has_beads(main_root)
+
+    dolt_ready = False
+    dolt_warning: str | None = None
+    if bd_available:
+        dolt_ready = ensure_dolt_server(bd_cwd)
+        if not dolt_ready:
+            dolt_warning = (
+                "Dolt server could not be started — bd data unavailable. "
+                "Run 'bd dolt start' manually from the project root and retry."
+            )
+
+    report = {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "project_root": str(root),
+        "bd_available": bd_available,
+        "bd_closed_issues": gather_bd_closed(bd_cwd) if dolt_ready else [],
+        "bd_memories": gather_bd_memories(bd_cwd) if dolt_ready else [],
+        "merged_prs": gather_merged_prs(root, since_n),
+        "recent_commits": gather_recent_commits(root, since_n),
+        "docs_drift": gather_docs_drift(root, since_n),
+    }
+
+    if dolt_warning:
+        report["warnings"] = [dolt_warning]
+
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == "__main__":
+    main()