xtrm-tools 2.4.1 → 2.4.2

package/skills/sync-docs-workspace/iteration-3/eval-sprint-closeout/without_skill/run-1/grading.json

@@ -0,0 +1,90 @@
+{
+  "expectations": [
+    {
+      "text": "Ran context_gatherer.py and reported bd closed issues or merged PRs with specific data",
+      "passed": false,
+      "evidence": "The agent never ran context_gatherer.py. It gathered context using raw git commands (git log --oneline --merges, git diff --stat 10d6433..HEAD). It did report specific merged PRs (#111, #110, #109) with descriptions, but the script was not used. The expectation requires the specific script to be invoked, not just the outcome data to be present."
+    },
+    {
+      "text": "Ran doc_structure_analyzer.py and cited its structured output (STALE, EXTRACTABLE, MISSING, etc.)",
+      "passed": false,
+      "evidence": "No mention of doc_structure_analyzer.py anywhere in the output. The structured output categories (STALE, EXTRACTABLE, MISSING) never appear. The agent assessed doc staleness manually by reading files and comparing with git history."
+    },
+    {
+      "text": "Detected the CHANGELOG version gap (package.json v2.4.0 vs CHANGELOG v2.0.0)",
+      "passed": false,
+      "evidence": "The output notes 'CHANGELOG.md (contains full history through v2.0.0)' and references the codebase being at v2.4.0, but the agent concluded CHANGELOG was 'accurate' and listed it under 'No Changes Needed'. It did not explicitly frame this as a version gap between package.json (v2.4.0) and CHANGELOG (v2.0.0), and it did not flag it as an issue requiring action. The gap was effectively missed because the agent treated the [Unreleased] section as sufficient coverage."
+    },
+    {
+      "text": "Named at least one concrete next step with a specific file or action",
+      "passed": true,
+      "evidence": "The Observations section states: 'The CHANGELOG [Unreleased] section is still empty \u2014 it should capture the post-v2.4.0 sprint work (global-first arch, guard-rules centralization, Pi drift checks, xtrm init project detection) before the next release.' This identifies a specific file (CHANGELOG.md), a specific section ([Unreleased]), and concrete content items to add."
+    }
+  ],
+  "summary": {
+    "passed": 1,
+    "failed": 3,
+    "total": 4,
+    "pass_rate": 0.25
+  },
+  "execution_metrics": {
+    "tool_calls": {},
+    "total_tool_calls": 0,
+    "total_steps": 0,
+    "errors_encountered": 0,
+    "output_chars": 3172,
+    "transcript_chars": 0
+  },
+  "timing": {
+    "executor_duration_seconds": 217.1,
+    "grader_duration_seconds": 0.0,
+    "total_duration_seconds": 217.1
+  },
+  "claims": [
+    {
+      "claim": "3 PRs merged in the most recent sprint: #111, #110, #109",
+      "type": "factual",
+      "verified": true,
+      "evidence": "Consistent with git log output cited in the result and with the repo's commit history (PR #111 referenced in CLAUDE.md recent commits section)"
+    },
+    {
+      "claim": "CHANGELOG.md is accurate and no changes are needed to it",
+      "type": "quality",
+      "verified": false,
+      "evidence": "The agent says CHANGELOG 'contains full history through v2.0.0' and the codebase is at v2.4.0. This means v2.1.0 through v2.4.0 entries are missing from CHANGELOG \u2014 a significant gap that contradicts the 'accurate' verdict. The [Unreleased] section does not substitute for missing versioned entries."
+    },
+    {
+      "claim": "XTRM-GUIDE.md required no changes as it was updated by sprint commits",
+      "type": "quality",
+      "verified": false,
+      "evidence": "The claim is plausible given commit f8e37f9, but the agent did not run doc_structure_analyzer.py or any systematic staleness check against XTRM-GUIDE.md \u2014 it relied on reading the file and comparing manually. Cannot fully verify without the script output."
+    },
+    {
+      "claim": "README was 'about 1.5 versions behind HEAD'",
+      "type": "factual",
+      "verified": true,
+      "evidence": "README said v2.3.0 while codebase was at v2.4.0 with unreleased post-v2.4.0 work on top \u2014 the characterization is reasonable given the 8 changes fixed."
+    }
+  ],
+  "user_notes_summary": {
+    "uncertainties": [],
+    "needs_review": [],
+    "workarounds": []
+  },
+  "eval_feedback": {
+    "suggestions": [
+      {
+        "assertion": "Ran context_gatherer.py and reported bd closed issues or merged PRs with specific data",
+        "reason": "This assertion conflates two things: running the specific script AND reporting specific PR data. An agent that skips the script but manually finds the same PR data would fail on process but produce similar outputs. The eval would be stronger if split: one assertion for script invocation (verifiable from transcript tool calls) and one for PR data quality (verifiable from output content)."
+      },
+      {
+        "assertion": "Detected the CHANGELOG version gap (package.json v2.4.0 vs CHANGELOG v2.0.0)",
+        "reason": "The expectation is well-targeted, but the bar should be higher: not just 'detected' but 'flagged as a problem requiring action'. The agent did notice CHANGELOG goes to v2.0.0 while the code is at v2.4.0, yet concluded it was accurate. An assertion that checks whether the gap was identified as a documentation deficiency (not just noted in passing) would be more discriminating."
+      },
+      {
+        "reason": "No assertion covers output quality for the README edits that were actually made \u2014 the primary work product of this run. The agent claims to have fixed 8 categories of README issues, but no expectation checks whether those changes are correct, complete, or even present in the file. This is the largest unguarded outcome."
+      }
+    ],
+    "overall": "The evals focus on process steps (run script X, detect gap Y) but miss the primary output (README changes). The CHANGELOG gap assertion is good but needs tighter framing. The script-invocation assertions are fragile without transcript access to verify tool calls."
+  }
+}

package/skills/sync-docs-workspace/iteration-3/eval-sprint-closeout/without_skill/run-1/timing.json

@@ -0,0 +1,5 @@
+{
+  "total_tokens": 61815,
+  "duration_ms": 217061,
+  "total_duration_seconds": 217.1
+}

package/skills/updating-service-skills/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: updating-service-skills
+description: >-
+  Detect implementation drift and sync expert persona documentation.
+  Activates automatically via PostToolUse hook when files in a registered
+  service territory are modified. Use when a skill's documentation has
+  fallen behind the actual implementation.
+hooks:
+  PostToolUse:
+    - matcher: "Write|Edit"
+      hooks:
+        - type: command
+          command: "python3 \"$CLAUDE_PROJECT_DIR/.claude/skills/updating-service-skills/scripts/drift_detector.py\" check-hook"
+          timeout: 10
+allowed-tools: Read, Write, Grep, Glob
+---
+
+# Updating Service Skills
+
+## Role: The Librarian
+
+You are the **Service Skills Librarian**. Your job is to keep expert persona
+documentation in sync with the actual implementation as the codebase evolves.
+
+---
+
+## Automatic Drift Detection
+
+After any `Write` or `Edit` operation, the `PostToolUse` hook runs
+`drift_detector.py check-hook`. It reads the modified file path from stdin JSON
+and checks whether it falls within a registered service territory.
+
+If drift is detected, you will see this in your context:
+
+```
+[Skill Sync]: Implementation drift detected in 'db-expert'.
+File 'src/db/users.ts' was modified.
+Use '/updating-service-skills' to sync the Database Expert documentation.
+```
+
+---
+
+## Manual Sync Process
+
+### Step 1 — Scan for all drift
+
+```bash
+python3 "$CLAUDE_PROJECT_DIR/.claude/skills/updating-service-skills/scripts/drift_detector.py" scan
+```
+
+### Step 2 — Read the current skill
+
+```
+Read: .claude/skills/<service-id>/SKILL.md
+```
+
+### Step 3 — Analyze changes using Serena tools
+
+Use Serena LSP tools (not raw file reads) to understand what changed:
+
+```
+get_symbols_overview(<modified-file>, depth=1)
+find_symbol(<changed-function>, include_body=True)
+search_for_pattern("<new-pattern>")
+```
+
+### Step 4 — Update the skill documentation
+
+- Add new patterns or conventions discovered
+- Update Failure Modes table if new exception handlers added
+- Update log patterns in `scripts/log_hunter.py` if new log strings found
+- Update territory patterns in `service-registry.json` if scope expanded
+- Preserve `<!-- SEMANTIC_START --> ... <!-- SEMANTIC_END -->` blocks
+
+### Step 5 — Mark as synced
+
+```bash
+python3 "$CLAUDE_PROJECT_DIR/.claude/skills/updating-service-skills/scripts/drift_detector.py" \
+  sync <service-id>
+```
+
+---
+
+## Drift Scenarios
+
+### New error pattern added to codebase
+
+1. `search_for_pattern("raise.*New.*Error|logger.error.*new")` to find it
+2. Add to `scripts/log_hunter.py` PATTERNS list with correct severity
+3. Update Troubleshooting table in SKILL.md
+
+### Territory expanded (new directory added)
+
+1. Check if current glob patterns in `service-registry.json` cover new files
+2. If not, update `territory` array in `service-registry.json`
+3. Sync timestamp
+
+### Major refactor changes conventions
+
+1. `get_symbols_overview` on all changed files
+2. Rewrite relevant Guidelines section in SKILL.md
+3. Update health_probe.py if table structure or ports changed
+
+---
+
+## Tool Restrictions
+
+Write to:
+- ✅ `.claude/skills/*/SKILL.md` — skill documentation updates
+- ✅ `.claude/skills/service-registry.json` — territory and sync timestamp updates
+
+Avoid:
+- ❌ Modify source code (read-only access to service territories)
+- ❌ Delete skills or registry entries
+
+---
+
+## Sync Output Format
+
+```
+✅ Skill Synced: `<service-id>`
+
+Updated:
+- log_hunter.py: added 2 new patterns from exception handlers
+- SKILL.md: Failure Modes table updated with OAuth expiry scenario
+- Territory: unchanged
+
+Next sync: triggers on next modification to <territory-patterns>
+```
+
+---
+
+## Related Skills
+
+- `/using-service-skills` — Discover and activate expert personas
+- `/creating-service-skills` — Scaffold new expert personas

package/skills/updating-service-skills/scripts/drift_detector.py

@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+"""
+Drift detector — PostToolUse hook for updating-service-skills.
+
+Checks whether a modified file belongs to a registered service territory.
+If so, notifies Claude to sync the skill documentation.
+
+Configured via updating-service-skills/SKILL.md frontmatter hooks:
+  PostToolUse → matcher: "Write|Edit" → command: drift_detector.py check-hook
+
+Subcommands:
+  check-hook          Read file_path from stdin JSON (PostToolUse hook mode)
+  check <file>        Check a specific file path from CLI
+  sync <service-id>   Mark service as synced (update registry timestamp)
+  scan                Scan all territories for drifted services
+"""
+
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+# Bootstrap lives in creating-service-skills — shared utility
+BOOTSTRAP_DIR = Path(__file__).parent.parent.parent / "creating-service-skills" / "scripts"
+sys.path.insert(0, str(BOOTSTRAP_DIR))
+
+from bootstrap import (  # noqa: E402
+    RootResolutionError,
+    find_service_for_path,
+    get_project_root,
+    get_service,
+    load_registry,
+    save_registry,
+)
+
+
+def check_drift(file_path: str, project_root: str | None = None) -> dict:
+    """Check if a file change causes drift in any registered service territory."""
+    if project_root is None:
+        try:
+            project_root = get_project_root()
+        except RootResolutionError:
+            return {"drift": False, "reason": "Cannot resolve project root"}
+
+    # Normalize to relative path
+    fp = Path(file_path)
+    if fp.is_absolute():
+        try:
+            fp = fp.relative_to(project_root)
+            file_path = str(fp)
+        except ValueError:
+            pass
+
+    service_id = find_service_for_path(file_path, project_root)
+    if not service_id:
+        return {"drift": False, "reason": "No service owns this file"}
+
+    service = get_service(service_id, project_root)
+    if not service:
+        return {"drift": False, "reason": "Service not found in registry"}
+
+    return {
+        "drift": True,
+        "service_id": service_id,
+        "service_name": service.get("name", service_id),
+        "skill_path": service.get("skill_path"),
+        "last_sync": service.get("last_sync", ""),
+        "file_path": file_path,
+        "message": (
+            f"[Skill Sync]: Implementation drift detected in '{service_id}'. "
+            f"File '{file_path}' was modified. "
+            f"Use '/updating-service-skills' to sync the {service.get('name', service_id)} documentation."
+        ),
+    }
+
+
+def check_drift_from_hook_stdin() -> None:
+    """
+    PostToolUse hook mode: reads file_path from stdin JSON.
+
+    The PostToolUse hook delivers stdin JSON with shape:
+      {"tool_name": "Write", "tool_input": {"file_path": "/abs/path"}, ...}
+
+    Outputs drift message to stdout (shown to Claude via PostToolUse additionalContext).
+    Exits 0 always — drift detection is advisory, never blocking.
+    """
+    try:
+        data = json.load(sys.stdin)
+    except (json.JSONDecodeError, EOFError):
+        sys.exit(0)
+
+    tool_input = data.get("tool_input", {})
+    file_path = tool_input.get("file_path", "")
+
+    if not file_path:
+        sys.exit(0)
+
+    result = check_drift(file_path)
+
+    if result.get("drift"):
+        # Output additionalContext for Claude via PostToolUse JSON format
+        output = {
+            "hookSpecificOutput": {
+                "hookEventName": "PostToolUse",
+                "additionalContext": result["message"],
+            }
+        }
+        print(json.dumps(output))
+
+    sys.exit(0)
+
+
+def update_sync_time(service_id: str, project_root: str | None = None) -> bool:
+    """Update last_sync timestamp for a service in the registry."""
+    try:
+        registry = load_registry(project_root)
+    except Exception:
+        return False
+
+    if "services" not in registry or service_id not in registry["services"]:
+        return False
+
+    registry["services"][service_id]["last_sync"] = (
+        datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+    )
+
+    try:
+        save_registry(registry, project_root)
+        return True
+    except Exception:
+        return False
+
+
+def scan_drift(project_root: str | None = None) -> list[dict]:
+    """Scan all territories and identify services with files modified since last sync."""
+    if project_root is None:
+        try:
+            project_root = get_project_root()
+        except RootResolutionError:
+            return []
+
+    root = Path(project_root)
+    registry = load_registry(project_root)
+    drifted: list[dict] = []
+
+    for service_id, service in registry.get("services", {}).items():
+        last_sync_str = service.get("last_sync", "")
+        if not last_sync_str:
+            continue
+        try:
+            sync_time = datetime.fromisoformat(last_sync_str.replace("Z", "+00:00"))
+        except ValueError:
+            continue
+
+        for pattern in service.get("territory", []):
+            for fp in root.glob(pattern):
+                if fp.is_file():
+                    try:
+                        mtime = datetime.fromtimestamp(fp.stat().st_mtime, tz=timezone.utc)
+                        if mtime > sync_time:
+                            drifted.append(
+                                {
+                                    "service_id": service_id,
+                                    "service_name": service.get("name", service_id),
+                                    "file_path": str(fp.relative_to(root)),
+                                    "last_sync": last_sync_str,
+                                }
+                            )
+                    except (OSError, ValueError):
+                        continue
+
+    return drifted
+
+
+def main() -> None:
+    if len(sys.argv) < 2:
+        print("Usage: python drift_detector.py <command> [args...]")
+        print("  check-hook          — Read file from stdin JSON (PostToolUse hook mode)")
+        print("  check <file>        — Check a specific file path")
+        print("  sync <service-id>   — Mark service as synced")
+        print("  scan                — Scan all territories for drift")
+        sys.exit(1)
+
+    cmd = sys.argv[1]
+
+    if cmd == "check-hook":
+        check_drift_from_hook_stdin()
+
+    elif cmd == "check" and len(sys.argv) > 2:
+        result = check_drift(sys.argv[2])
+        if result.get("drift"):
+            print(result["message"])
+        else:
+            print(f"No drift: {result.get('reason', 'OK')}")
+
+    elif cmd == "sync" and len(sys.argv) > 2:
+        service_id = sys.argv[2]
+        if update_sync_time(service_id):
+            print(f"✓ Synced: {service_id}")
+        else:
+            print(f"✗ Failed to sync: {service_id}")
+            sys.exit(1)
+
+    elif cmd == "scan":
+        drifted = scan_drift()
+        if drifted:
+            print(f"Found {len(drifted)} drifted service(s):")
+            for item in drifted:
+                print(
+                    f"  {item['service_id']}: {item['file_path']} "
+                    f"(last sync: {item['last_sync']})"
+                )
+        else:
+            print("No drift detected.")
+
+    else:
+        print(f"Unknown command: {cmd}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()