@cleocode/skills 2026.5.81 → 2026.5.83

package/skills/ct-grade-v2-1/agents/blind-comparator.md

@@ -1,157 +0,0 @@
-# Blind Comparator Agent
-
-You are a blind comparator for CLEO behavioral evaluation. You evaluate two outputs — labeled only as **Output A** and **Output B** — without knowing which configuration, interface, or scenario produced them.
-
-Your job is to produce an objective, evidence-based comparison in `comparison.json` format.
-
-## Critical Rules
-
-1. **You do NOT know and MUST NOT speculate** about which output came from MCP vs CLI, or which scenario variant was used.
-2. **Judge on observable output quality only**: correctness, completeness, protocol adherence, efficiency.
-3. **Be specific**: every score must have evidence from the actual outputs.
-4. **Score independently first**, then declare a winner.
-
-## Inputs
-
-You will receive:
-- `OUTPUT_A_PATH`: Path to arm A's output files (grade.json, operations.jsonl)
-- `OUTPUT_B_PATH`: Path to arm B's output files (grade.json, operations.jsonl)
-- `SCENARIO`: Which grade scenario was run (for rubric context)
-- `OUTPUT_PATH`: Where to write comparison.json
-
-## Evaluation Dimensions
-
-For each output, assess:
-
-### 1. Grade Score Accuracy (0-5 pts each)
-- Does the session score reflect the actual operations executed?
-- Are flags appropriate for the violations observed?
-- Is the score consistent with the evidence in the grade result?
-
-### 2. Protocol Adherence (0-5 pts each)
-- Were all required operations for the scenario executed?
-- Were operations in the correct order?
-- Were operations well-formed (descriptions provided, params complete)?
-
-### 3. Efficiency (0-5 pts each)
-- Did the execution use the minimal necessary operations?
-- Was `tasks.find` preferred over `tasks.list`?
-- Were redundant calls avoided?
-
-### 4. Error Handling (0-5 pts each)
-- Were errors (if any) properly recovered from?
-- Were no unnecessary errors triggered?
-
-## Process
-
-1. Read `grade.json` from both output dirs
-2. Read `operations.jsonl` from both output dirs
-3. Score each dimension for A and B independently
-4. Sum scores: content_score = (grade_accuracy + protocol_adherence) / 2, structure_score = (efficiency + error_handling) / 2
-5. Declare winner (or tie if within 0.5 points)
-6. Write comparison.json
-
-## Output Format
-
-Write `comparison.json` to `OUTPUT_PATH`:
-
-```json
-{
-  "winner": "A",
-  "reasoning": "Output A demonstrated complete protocol adherence with all 10 required operations executed in correct order. Output B missed the session.list-before-task-ops ordering, reducing its S1 score.",
-  "rubric": {
-    "A": {
-      "content": {
-        "grade_score_accuracy": 5,
-        "protocol_adherence": 5
-      },
-      "structure": {
-        "efficiency": 4,
-        "error_handling": 5
-      },
-      "content_score": 5.0,
-      "structure_score": 4.5,
-      "overall_score": 9.5
-    },
-    "B": {
-      "content": {
-        "grade_score_accuracy": 3,
-        "protocol_adherence": 2
-      },
-      "structure": {
-        "efficiency": 4,
-        "error_handling": 5
-      },
-      "content_score": 2.5,
-      "structure_score": 4.5,
-      "overall_score": 7.0
-    }
-  },
-  "output_quality": {
-    "A": {
-      "score": 9,
-      "strengths": ["All scenario operations present", "Correct ordering", "Descriptions on all tasks"],
-      "weaknesses": ["Slightly verbose operation params"]
-    },
-    "B": {
-      "score": 7,
-      "strengths": ["Efficient operation count", "Good error recovery"],
-      "weaknesses": ["session.list came after first task op (-10 S1)", "No admin.help call (-10 S5)"]
-    }
-  },
-  "grade_comparison": {
-    "A": {
-      "total_score": 95,
-      "grade": "A",
-      "flags": []
-    },
-    "B": {
-      "total_score": 75,
-      "grade": "B",
-      "flags": ["session.list called after task ops", "No admin.help or skill lookup calls"]
-    }
-  },
-  "expectation_results": {
-    "A": {
-      "passed": 5,
-      "total": 5,
-      "pass_rate": 1.0,
-      "details": [
-        {"text": "session.list before any task op", "passed": true},
-        {"text": "session.end called", "passed": true},
-        {"text": "tasks.find used for discovery", "passed": true},
-        {"text": "admin.help called", "passed": true},
-        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
-      ]
-    },
-    "B": {
-      "passed": 3,
-      "total": 5,
-      "pass_rate": 0.60,
-      "details": [
-        {"text": "session.list before any task op", "passed": false},
-        {"text": "session.end called", "passed": true},
-        {"text": "tasks.find used for discovery", "passed": true},
-        {"text": "admin.help called", "passed": false},
-        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
-      ]
-    }
-  }
-}
-```
-
-## Tie Handling
-
-If overall scores are within 0.5 points, declare `"winner": "tie"` and note both performed equivalently.
-
-## Final Summary
-
-After writing comparison.json, output:
-```
-WINNER: <A|B|tie>
-SCORE_A: <overall>
-SCORE_B: <overall>
-GRADE_A: <letter> (<total>/100)
-GRADE_B: <letter> (<total>/100)
-FILE: <comparison.json path>
-```

package/skills/ct-grade-v2-1/agents/scenario-runner.md

@@ -1,160 +0,0 @@
-# Scenario Runner Agent
-
-You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the CLI, capture the audit trail, and grade the resulting session.
-
-## Inputs
-
-You will receive:
-- `SCENARIO`: Which scenario to run (s1|s2|s3|s4|s5|s6|s7|s8|s9|s10)
-- `OUTPUT_DIR`: Where to write results
-- `PROJECT_DIR`: Path to the CLEO project (for cleo-dev --cwd)
-- `RUN_NUMBER`: Integer (1, 2, 3...) for repeated runs
-
-## Execution Protocol
-
-### Step 1: Record start time
-
-Note the ISO timestamp before any operations.
-
-### Step 2: Start a graded session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> session start --grade --name "grade-<SCENARIO>-run<RUN>" --scope global
-```
-
-Save the returned `sessionId`.
-
-If this fails (DB migration error, ENOENT, or non-zero exit):
-- Write `grade.json: { "error": "DB_UNAVAILABLE", "totalScore": null }`
-- Write `timing.json: { "error": "DB_UNAVAILABLE", "total_tokens": null, "duration_ms": null, "scenario": "<SCENARIO>", "run": <RUN_NUMBER>, "interface": "cli", "executor_start": "<ISO>", "executor_end": "<ISO>" }`
-- Output: `SESSION_START_FAILED: DB_UNAVAILABLE`
-- Stop. Do NOT abort silently.
-
-### Step 3: Execute scenario operations
-
-Follow the exact operation sequence from the scenario playbook. All operations use the CLI.
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> find --status active
-```
-
-Scenario sequences are in [../references/playbook-v2.md](../references/playbook-v2.md). Execute the operations in order. Do NOT skip operations — each one contributes to the grade.
-
-### Step 4: End the session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> session end
-```
-
-### Step 5: Grade the session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> check grade --session "<saved-id>"
-```
-
-Save the full GradeResult JSON.
-
-### Step 6: Capture operations log
-
-Record every operation you executed as a JSONL file. Each line:
-```json
-{"seq": 1, "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "cli", "timestamp": "..."}
-```
-
-### Step 7: Write output files
-
-Write to `<OUTPUT_DIR>/<SCENARIO>/arm-<INTERFACE>/`:
-
-**grade.json** — The GradeResult from check.grade:
-```json
-{
-  "sessionId": "...",
-  "totalScore": 85,
-  "maxScore": 100,
-  "dimensions": {...},
-  "flags": [...],
-  "entryCount": 12
-}
-```
-
-**operations.jsonl** — One JSON object per line, each operation executed.
-
-**timing.json** — Fill in what you can; orchestrator fills `total_tokens` and `duration_ms`:
-```json
-{
-  "scenario": "<SCENARIO>",
-  "run": <RUN_NUMBER>,
-  "interface": "cli",
-  "session_id": "<session-id>",
-  "executor_start": "<ISO>",
-  "executor_end": "<ISO>",
-  "executor_duration_seconds": 0,
-  "token_usage_id": "<id from admin.token response>",
-  "total_tokens": null,
-  "duration_ms": null
-}
-```
-
-Note: `total_tokens` and `duration_ms` are filled by the orchestrator from the task completion notification — you cannot read them yourself.
-
-### Step 8: Record token exchange (mandatory for token_usage table)
-
-After receiving the grade result, record the exchange to persist token measurements:
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> admin token record --session "<session-id>" --domain admin --operation grade --metadata '{"scenario":"<SCENARIO>","run":<RUN_NUMBER>}'
-```
-
-Save the returned `id` as `token_usage_id` in timing.json.
-
-## Quick Reference — Scenarios
-
-| Scenario | Name | Key Domains | Target Score |
-|----------|------|-------------|--------------|
-| s1 | Session Discipline | session, tasks | S1=20, S2=15+ |
-| s2 | Task Hygiene | tasks, session | S3=20, S1=20 |
-| s3 | Error Recovery | tasks, session | S4=20 |
-| s4 | Full Lifecycle | tasks, session, admin | All dims 15+ |
-| s5 | Multi-Domain Analysis | tasks, admin, pipeline | S5=15+ |
-| s6 | Memory Observe & Recall | memory, session | S5=15+, S2=15+ |
-| s7 | Decision Continuity | memory, session | S1=20, S5=15+ |
-| s8 | Pattern & Learning | memory, session | S2=15+, S5=15+ |
-| s9 | NEXUS Cross-Project | nexus, session, admin | S5=20, S1=20 |
-| s10 | Full System Throughput | all 8 domains | S2=15+, S5=15+ |
-
-## Scenario Key Operations
-
-| Scenario | Key Operations | S1 | S2 | S3 | S4 | S5 |
-|---|---|---|---|---|---|---|
-| s1 | session.list, tasks.find, tasks.show, session.end | ✓ | ✓ | — | — | partial |
-| s2 | session.list, tasks.find, tasks.add×2, session.end | ✓ | — | ✓ | — | — |
-| s3 | session.list, tasks.show (E_NOT_FOUND), tasks.find (recover), tasks.add, session.end | ✓ | — | ✓ | ✓ | — |
-| s4 | session.list, admin.help, tasks.find, tasks.show, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
-| s5 | session.list, admin.help, tasks.find (parent filter), tasks.show, session.context.drift, session.decision.log, session.record.decision, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
-| s6 | memory.observe, memory.find, memory.timeline, memory.fetch, session.end | ✓ | ✓ | — | — | ✓ |
-| s7 | memory.decision.store, memory.decision.find, memory.find, memory.fetch, session.end | ✓ | — | — | — | ✓ |
-| s8 | memory.pattern.store, memory.learning.store, memory.pattern.find, memory.learning.find, session.end | — | ✓ | — | — | ✓ |
-| s9 | nexus.status, nexus.list, nexus.show, admin.dash, session.end | ✓ | — | — | — | ✓ |
-| s10 | session.list, admin.help, tasks.find, memory.find, nexus.status, pipeline.stage.status, admin.health, tools.skill.list, memory.observe, session.end | ✓ | ✓ | — | — | ✓ |
-
-## Anti-patterns to Avoid
-
-Do NOT do these during scenario execution — they will lower the grade intentionally only if you are running the anti-pattern variant:
-- Calling `tasks.list` instead of `tasks.find` for discovery
-- Skipping `session.list` at the start
-- Creating tasks without descriptions
-- Ignoring `E_NOT_FOUND` errors without recovery lookup
-- Never calling `admin.help`
-
-## Output
-
-When complete, summarize:
-```
-SCENARIO: <id>
-RUN: <n>
-SESSION_ID: <id>
-TOTAL_SCORE: <n>/100
-GRADE: <letter>
-FLAGS: <count>
-FILES_WRITTEN: <list>
-```

package/skills/ct-grade-v2-1/evals/evals.json

@@ -1,74 +0,0 @@
-[
-  {
-    "id": "eval-001",
-    "description": "Grade a session — verify grading pipeline returns a valid GradeResult",
-    "prompt": "Start a graded session, run query session list and admin dash, end session, then grade it",
-    "expectations": [
-      "Grade operation returns success: true",
-      "totalScore is a number 0-100",
-      "dimensions has 5 entries each with score and max",
-      "flags is an array"
-    ]
-  },
-  {
-    "id": "eval-002",
-    "description": "Session discipline — session.list before task ops scores S1=20",
-    "prompt": "Run scenario S1 and verify session discipline dimension is 20/20",
-    "expectations": [
-      "S1 Session Discipline score = 20",
-      "session.list was called before any task operation",
-      "session.end was called",
-      "No protocol flags"
-    ]
-  },
-  {
-    "id": "eval-003",
-    "description": "Task efficiency — tasks.find used (not tasks.list) scores S2>=15",
-    "prompt": "Run tasks.find query and verify efficiency score is 15 or higher",
-    "expectations": [
-      "S2 Task Efficiency score >= 15",
-      "tasks.find was used instead of tasks.list",
-      "No TASK_LIST_USED flag"
-    ]
-  },
-  {
-    "id": "eval-004",
-    "description": "Task hygiene — task add with description scores S3=20",
-    "prompt": "Add a task with both title and description, verify hygiene score is 20",
-    "expectations": [
-      "S3 Task Hygiene score = 20",
-      "Task was created with non-empty description",
-      "No MISSING_DESCRIPTION flag"
-    ]
-  },
-  {
-    "id": "eval-005",
-    "description": "Protocol adherence — following CLEO workflow scores S4>=15",
-    "prompt": "Follow the complete CLEO session workflow and verify protocol adherence",
-    "expectations": [
-      "S4 Protocol Adherence score >= 15",
-      "Session started before task work",
-      "Session ended after task work"
-    ]
-  },
-  {
-    "id": "eval-006",
-    "description": "MCP gateway — MCP-sourced ops score S5>=15",
-    "prompt": "Use MCP interface for all operations and verify gateway score is 15 or higher",
-    "expectations": [
-      "S5 MCP Gateway score >= 15",
-      "Operations sourced from MCP (not CLI)",
-      "audit_log shows gateway=query or gateway=mutate with source=mcp"
-    ]
-  },
-  {
-    "id": "eval-007",
-    "description": "Memory recall — observe then find retrieves the observation",
-    "prompt": "Run scenario S6: observe a fact then find it via memory.find",
-    "expectations": [
-      "memory.observe succeeds and returns an ID",
-      "memory.find with matching query returns the observation",
-      "Grade total score >= 60"
-    ]
-  }
-]

package/skills/ct-grade-v2-1/grade-viewer/build_op_stats.py

@@ -1,174 +0,0 @@
-#!/usr/bin/env python3
-"""
-build_op_stats.py — Aggregate operations.jsonl files from grade runs into per-operation statistics.
-
-Reads all operations.jsonl files under --grade-runs-dir and computes per-operation stats
-split by interface (mcp/cli). Output is a JSON object keyed by "domain.operation".
-
-Usage:
-    python build_op_stats.py [options]
-
-Options:
-    --grade-runs-dir PATH    Directory containing grade run subdirectories
-                             (default: .cleo/metrics/grade-runs relative to cwd)
-    --output PATH            Output JSON file path
-                             (default: .cleo/metrics/per_operation_stats.json)
-    --pretty                 Pretty-print JSON output (default: compact)
-    --verbose                Print progress to stderr
-
-Output format (per key "domain.operation"):
-{
-    "mcp_calls": 42,
-    "cli_calls": 10,
-    "total_mcp_ms": 1234.5,
-    "total_cli_ms": 456.7,
-    "avg_mcp_ms": 29.4,
-    "avg_cli_ms": 45.7,
-    "runs_seen": 3
-}
-
-Also importable as a module:
-    from build_op_stats import compute_stats
-    stats = compute_stats(grade_runs_dir="/path/to/grade-runs")
-"""
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-
-def compute_stats(grade_runs_dir, verbose=False):
-    """
-    Aggregate operations.jsonl files under grade_runs_dir.
-
-    Returns dict keyed by "domain.operation" with accumulated stats.
-    """
-    runs_dir = Path(grade_runs_dir)
-    stats = {}
-    files_processed = 0
-    lines_processed = 0
-
-    if not runs_dir.exists():
-        if verbose:
-            print(f"[build_op_stats] Grade runs dir not found: {runs_dir}", file=sys.stderr)
-        return stats
-
-    for ops_file in sorted(runs_dir.rglob('operations.jsonl')):
-        files_processed += 1
-        if verbose:
-            print(f"[build_op_stats] Processing: {ops_file}", file=sys.stderr)
-
-        for line in ops_file.read_text(errors='replace').splitlines():
-            line = line.strip()
-            if not line:
-                continue
-            try:
-                entry = json.loads(line)
-            except json.JSONDecodeError:
-                continue
-
-            domain = entry.get('domain', 'unknown')
-            operation = entry.get('operation', 'unknown')
-            key = f"{domain}.{operation}"
-            interface = entry.get('interface', 'mcp')
-            duration = float(entry.get('duration_ms', 0) or 0)
-
-            if key not in stats:
-                stats[key] = {
-                    'mcp_calls': 0,
-                    'cli_calls': 0,
-                    'total_mcp_ms': 0.0,
-                    'total_cli_ms': 0.0,
-                    'avg_mcp_ms': 0.0,
-                    'avg_cli_ms': 0.0,
-                    'runs_seen': set(),
-                }
-
-            # Track which run directory this came from
-            # ops_file is e.g. .../grade-runs/run-20260308/s1/run-01/arm-mcp/operations.jsonl
-            # run_id is the first path component relative to runs_dir (e.g. "run-20260308")
-            run_id = ops_file.relative_to(runs_dir).parts[0]
-            stats[key]['runs_seen'].add(run_id)
-
-            if interface == 'cli':
-                stats[key]['cli_calls'] += 1
-                stats[key]['total_cli_ms'] += duration
-            else:
-                stats[key]['mcp_calls'] += 1
-                stats[key]['total_mcp_ms'] += duration
-
-            lines_processed += 1
-
-    # Compute averages and convert sets to counts
-    for key, v in stats.items():
-        mc = v['mcp_calls']
-        cc = v['cli_calls']
-        v['avg_mcp_ms'] = round(v['total_mcp_ms'] / mc, 2) if mc > 0 else 0.0
-        v['avg_cli_ms'] = round(v['total_cli_ms'] / cc, 2) if cc > 0 else 0.0
-        v['total_mcp_ms'] = round(v['total_mcp_ms'], 2)
-        v['total_cli_ms'] = round(v['total_cli_ms'], 2)
-        v['runs_seen'] = len(v['runs_seen'])
-
-    if verbose:
-        print(f"[build_op_stats] Processed {files_processed} files, {lines_processed} lines → {len(stats)} unique operations", file=sys.stderr)
-
-    return stats
-
-
-def find_cleo_dir(start='.'):
-    """Walk up from start to find directory containing .cleo/tasks.db."""
-    p = Path(start).resolve()
-    while p != p.parent:
-        if (p / '.cleo' / 'tasks.db').exists():
-            return p
-        p = p.parent
-    return Path(start).resolve()
-
-
-def main():
-    parser = argparse.ArgumentParser(
-        description='Aggregate grade run operations.jsonl files into per-operation stats.'
-    )
-    parser.add_argument(
-        '--grade-runs-dir',
-        default=None,
-        help='Directory containing grade run subdirectories (default: .cleo/metrics/grade-runs)'
-    )
-    parser.add_argument(
-        '--output',
-        default=None,
-        help='Output JSON path (default: .cleo/metrics/per_operation_stats.json)'
-    )
-    parser.add_argument(
-        '--pretty',
-        action='store_true',
-        help='Pretty-print JSON output'
-    )
-    parser.add_argument(
-        '--verbose',
-        action='store_true',
-        help='Print progress to stderr'
-    )
-    args = parser.parse_args()
-
-    workspace = find_cleo_dir('.')
-
-    grade_runs_dir = args.grade_runs_dir or str(workspace / '.cleo' / 'metrics' / 'grade-runs')
-    output_path = args.output or str(workspace / '.cleo' / 'metrics' / 'per_operation_stats.json')
-
-    stats = compute_stats(grade_runs_dir, verbose=args.verbose)
-
-    indent = 2 if args.pretty else None
-    output_json = json.dumps(stats, indent=indent)
-
-    out = Path(output_path)
-    out.parent.mkdir(parents=True, exist_ok=True)
-    out.write_text(output_json)
-
-    print(f"Wrote {len(stats)} operation stats to {output_path}")
-    return 0
-
-
-if __name__ == '__main__':
-    sys.exit(main())

package/skills/ct-grade-v2-1/grade-viewer/eval-analysis.json

@@ -1,41 +0,0 @@
-{
-  "total_grades": 31,
-  "score_distribution": {
-    "F (0)": 7,
-    "D (45-59)": 10,
-    "C (60-74)": 5,
-    "B (75-89)": 8,
-    "A (90+)": 1
-  },
-  "score_stats": {
-    "mean": 64.6,
-    "min": 50,
-    "max": 95,
-    "grades_with_data": 24,
-    "zero_score_count": 7
-  },
-  "dimension_averages": {
-    "sessionDiscipline": 5.8,
-    "discoveryEfficiency": 9.0,
-    "taskHygiene": 15.4,
-    "errorProtocol": 15.3,
-    "disclosureUse": 4.5
-  },
-  "flag_frequency": {
-    "No admin.help calls": 21,
-    "session.list never called": 18,
-    "No MCP query calls": 13,
-    "session.end never called": 12,
-    "No audit entries": 7,
-    "tasks.list used (prefer find)": 5,
-    "Subtasks without exists check": 1,
-    "Duplicate task creates": 1
-  },
-  "avg_audit_entries": 9.5,
-  "token_estimate": {
-    "avg_per_session_chars": 0,
-    "avg_per_session_tokens": 1425.0,
-    "method": "entry_count * 150 proxy",
-    "note": "OTEL not enabled; enable with CLAUDE_CODE_ENABLE_TELEMETRY=1 for real counts"
-  }
-}

package/skills/ct-grade-v2-1/grade-viewer/eval-report.md

@@ -1,37 +0,0 @@
-# CLEO Grade v2.1 — Comparative Analysis Report
-
-**Generated:** 2026-03-07 23:47 UTC  
-**Source:** `/tmp/ct-grade-eval`
-
-> **DEPRECATED**: This report was generated when MCP was still supported. MCP has been removed.
-> All operations now use the CLI exclusively. These results are retained for historical reference only.
-
----
-
-## Historical: MCP vs CLI Blind A/B Results
-
-**Overall winner: MCP**  
-
-| Metric | Value |
-|--------|-------|
-| Total runs | 3 |
-| MCP wins | 3 (100.0%) |
-| CLI wins | 0 (0.0%) |
-| Ties | 0 |
-| Avg token delta (MCP–CLI) | +416.0 tokens |
-| Interpretation | MCP uses more tokens on average |
-
-### Per-Operation Results
-
-| Operation | MCP wins | CLI wins | Ties | Token delta | MCP chars | CLI chars | MCP ms | CLI ms |
-|-----------|----------|----------|------|-------------|-----------|-----------|--------|--------|
-| `admin.version` **MCP** | 3 | 0 | 0 | +416t | 1664 | 0 | 930ms | 786ms |
-
-### Recommendations
-
-- **MCP adds significant token overhead.** Consider whether MCP envelope verbosity can be reduced for high-frequency operations.
-- **MCP output quality is consistently higher.** Reinforces MCP-first agent protocol recommendation.
-
----
-
-*Report generated by ct-grade v2.1 `generate_report.py`*