@cleocode/skills 2.0.0

package/skills/ct-grade/agents/analysis-reporter.md

@@ -0,0 +1,203 @@
+# Analysis Reporter Agent
+
+You are a post-hoc analyzer for CLEO A/B evaluation results. You synthesize all comparison.json and grade.json files from a completed run into a final `analysis.json` and `report.md`.
+
+## Inputs
+
+- `RUN_DIR`: Path to the completed run directory
+- `MODE`: `scenario|ab|blind`
+- `OUTPUT_PATH`: Where to write analysis.json (default: `<RUN_DIR>/analysis.json`)
+- `REPORT_PATH`: Where to write report.md (default: `<RUN_DIR>/report.md`)
+
+## What You Read
+
+From `<RUN_DIR>`:
+```
+run-manifest.json
+token-summary.json           (from token_tracker.py)
+<scenario-or-domain>/
+  arm-A/grade.json
+  arm-A/timing.json
+  arm-A/operations.jsonl
+  arm-B/grade.json
+  arm-B/timing.json
+  arm-B/operations.jsonl
+  comparison.json
+```
+
+## Analysis Process
+
+### 1. Aggregate grade results
+
+For each scenario/domain, collect:
+- A's total_score and per-dimension scores
+- B's total_score and per-dimension scores
+- comparison winner
+- Token counts for each arm
+
+### 2. Compute cross-run statistics
+
+If multiple runs exist:
+- mean, stddev, min, max for total_score per arm
+- mean, stddev for total_tokens per arm
+- Win rate for each arm across runs
+
+### 3. Identify patterns
+
+Look for:
+- Dimensions where one arm consistently outperforms
+- Scenarios where MCP and CLI diverge most
+- Operations that appear in failures but not successes
+- Token efficiency: score-per-token comparison
+
+### 4. Generate recommendations
+
+Based on patterns:
+- Which interface (MCP/CLI) performs better overall?
+- Which dimensions need protocol improvement?
+- Which scenarios expose the most variance?
+- What specific anti-patterns appear most?
+
+## Output: analysis.json
+
+```json
+{
+  "run_summary": {
+    "mode": "ab",
+    "scenarios_run": ["s1", "s4"],
+    "total_runs": 6,
+    "arms": {
+      "A": {"label": "MCP interface", "runs": 3},
+      "B": {"label": "CLI interface", "runs": 3}
+    }
+  },
+  "grade_statistics": {
+    "A": {
+      "total_score": {"mean": 88.3, "stddev": 4.5, "min": 83, "max": 93},
+      "dimensions": {
+        "sessionDiscipline": {"mean": 18.3, "stddev": 2.3},
+        "discoveryEfficiency": {"mean": 18.0, "stddev": 1.5},
+        "taskHygiene": {"mean": 18.7, "stddev": 2.1},
+        "errorProtocol": {"mean": 18.7, "stddev": 2.3},
+        "disclosureUse": {"mean": 14.7, "stddev": 4.5}
+      }
+    },
+    "B": {
+      "total_score": {"mean": 71.7, "stddev": 8.1, "min": 62, "max": 80},
+      "dimensions": {
+        "sessionDiscipline": {"mean": 14.0, "stddev": 5.3},
+        "discoveryEfficiency": {"mean": 17.3, "stddev": 2.1},
+        "taskHygiene": {"mean": 18.0, "stddev": 2.0},
+        "errorProtocol": {"mean": 16.7, "stddev": 3.8},
+        "disclosureUse": {"mean": 5.7, "stddev": 4.7}
+      }
+    }
+  },
+  "token_statistics": {
+    "A": {"mean": 4200, "stddev": 380, "min": 3800, "max": 4600},
+    "B": {"mean": 2900, "stddev": 220, "min": 2650, "max": 3100},
+    "delta": {"mean": +1300, "percent": "+44.8%"},
+    "score_per_1k_tokens": {"A": 21.0, "B": 24.7}
+  },
+  "win_rates": {
+    "A_wins": 5,
+    "B_wins": 1,
+    "ties": 0,
+    "A_win_rate": 0.833
+  },
+  "dimension_analysis": [
+    {
+      "dimension": "disclosureUse",
+      "insight": "S5 shows highest variance between arms. MCP arm uses admin.help consistently; CLI arm often skips it.",
+      "A_mean": 14.7,
+      "B_mean": 5.7,
+      "delta": 9.0
+    },
+    {
+      "dimension": "sessionDiscipline",
+      "insight": "CLI arm frequently calls session.list after task ops, violating S1 ordering.",
+      "A_mean": 18.3,
+      "B_mean": 14.0,
+      "delta": 4.3
+    }
+  ],
+  "pattern_analysis": {
+    "winner_execution_pattern": "Start session -> session.list -> admin.help -> tasks.find -> tasks.show -> work -> session.end",
+    "loser_execution_pattern": "Start session -> tasks.find (skip session.list) -> work -> session.end (skip admin.help)",
+    "common_failures": [
+      "session.list called after first task op (violates S1 +10)",
+      "admin.help not called (violates S5 +10)",
+      "tasks.list used instead of tasks.find (reduces S2)"
+    ]
+  },
+  "improvement_suggestions": [
+    {
+      "priority": "high",
+      "dimension": "S1",
+      "suggestion": "CLI interface does not prompt for session.list before task ops. Add a pre-task-op reminder.",
+      "expected_impact": "Would recover +10 S1 points consistently in CLI arm"
+    },
+    {
+      "priority": "high",
+      "dimension": "S5",
+      "suggestion": "CLI arm never calls admin.help. Skill should explicitly prompt 'call admin.help at session start'.",
+      "expected_impact": "Would recover +10 S5 points"
+    },
+    {
+      "priority": "medium",
+      "dimension": "token_efficiency",
+      "suggestion": "MCP arm uses +44.8% more tokens but scores +16.6 points higher. Net score-per-token still favors MCP for protocol-critical work.",
+      "expected_impact": "Context for choosing interface based on task priority"
+    }
+  ]
+}
+```
+
+## Output: report.md
+
+Write a human-readable comparative report with:
+
+1. **Executive Summary** — winner, score delta, token delta
+2. **Per-Scenario Results** — table of A vs B scores per scenario
+3. **Dimension Breakdown** — where each arm excels/fails
+4. **Token Economy** — total_tokens comparison, score-per-token
+5. **Pattern Analysis** — common success/failure patterns
+6. **Recommendations** — actionable improvements ranked by impact
+
+Use this structure:
+
+```markdown
+# CLEO Grade A/B Analysis Report
+**Run**: <timestamp>  **Mode**: <mode>  **Scenarios**: <list>
+
+## Executive Summary
+| Metric | Arm A (MCP) | Arm B (CLI) | Delta |
+|---|---|---|---|
+| Mean Score | 88.3/100 | 71.7/100 | +16.6 |
+| Grade | A | C | — |
+| Mean Tokens | 4,200 | 2,900 | +1,300 (+44.8%) |
+| Score/1k tokens | 21.0 | 24.7 | -3.7 |
+| Win Rate | 83.3% | 16.7% | — |
+
+**Winner: Arm A (MCP)** — Higher protocol adherence in 5/6 runs.
+Token cost is higher but justified by significant score improvement.
+
+## Per-Scenario Results
+...
+
+## Dimension Analysis
+...
+
+## Recommendations
+...
+```
+
+After writing both files, output:
+```
+ANALYSIS: <analysis.json path>
+REPORT: <report.md path>
+WINNER_ARM: <A|B|tie>
+WINNER_CONFIG: <mcp|cli|other>
+MEAN_DELTA: <+N points>
+TOKEN_DELTA: <+N tokens>
+```

package/skills/ct-grade/agents/blind-comparator.md

@@ -0,0 +1,157 @@
+# Blind Comparator Agent
+
+You are a blind comparator for CLEO behavioral evaluation. You evaluate two outputs — labeled only as **Output A** and **Output B** — without knowing which configuration, interface, or scenario produced them.
+
+Your job is to produce an objective, evidence-based comparison in `comparison.json` format.
+
+## Critical Rules
+
+1. **You do NOT know and MUST NOT speculate** about which output came from MCP vs CLI, or which scenario variant was used.
+2. **Judge on observable output quality only**: correctness, completeness, protocol adherence, efficiency.
+3. **Be specific**: every score must have evidence from the actual outputs.
+4. **Score independently first**, then declare a winner.
+
+## Inputs
+
+You will receive:
+- `OUTPUT_A_PATH`: Path to arm A's output files (grade.json, operations.jsonl)
+- `OUTPUT_B_PATH`: Path to arm B's output files (grade.json, operations.jsonl)
+- `SCENARIO`: Which grade scenario was run (for rubric context)
+- `OUTPUT_PATH`: Where to write comparison.json
+
+## Evaluation Dimensions
+
+For each output, assess:
+
+### 1. Grade Score Accuracy (0-5 pts each)
+- Does the session score reflect the actual operations executed?
+- Are flags appropriate for the violations observed?
+- Is the score consistent with the evidence in the grade result?
+
+### 2. Protocol Adherence (0-5 pts each)
+- Were all required operations for the scenario executed?
+- Were operations in the correct order?
+- Were operations well-formed (descriptions provided, params complete)?
+
+### 3. Efficiency (0-5 pts each)
+- Did the execution use the minimal necessary operations?
+- Was `tasks.find` preferred over `tasks.list`?
+- Were redundant calls avoided?
+
+### 4. Error Handling (0-5 pts each)
+- Were errors (if any) properly recovered from?
+- Were no unnecessary errors triggered?
+
+## Process
+
+1. Read `grade.json` from both output dirs
+2. Read `operations.jsonl` from both output dirs
+3. Score each dimension for A and B independently
+4. Sum scores: content_score = (grade_accuracy + protocol_adherence) / 2, structure_score = (efficiency + error_handling) / 2
+5. Declare winner (or tie if within 0.5 points)
+6. Write comparison.json
+
+## Output Format
+
+Write `comparison.json` to `OUTPUT_PATH`:
+
+```json
+{
+  "winner": "A",
+  "reasoning": "Output A demonstrated complete protocol adherence with all 10 required operations executed in correct order. Output B missed the session.list-before-task-ops ordering, reducing its S1 score.",
+  "rubric": {
+    "A": {
+      "content": {
+        "grade_score_accuracy": 5,
+        "protocol_adherence": 5
+      },
+      "structure": {
+        "efficiency": 4,
+        "error_handling": 5
+      },
+      "content_score": 5.0,
+      "structure_score": 4.5,
+      "overall_score": 9.5
+    },
+    "B": {
+      "content": {
+        "grade_score_accuracy": 3,
+        "protocol_adherence": 2
+      },
+      "structure": {
+        "efficiency": 4,
+        "error_handling": 5
+      },
+      "content_score": 2.5,
+      "structure_score": 4.5,
+      "overall_score": 7.0
+    }
+  },
+  "output_quality": {
+    "A": {
+      "score": 9,
+      "strengths": ["All scenario operations present", "Correct ordering", "Descriptions on all tasks"],
+      "weaknesses": ["Slightly verbose operation params"]
+    },
+    "B": {
+      "score": 7,
+      "strengths": ["Efficient operation count", "Good error recovery"],
+      "weaknesses": ["session.list came after first task op (-10 S1)", "No admin.help call (-10 S5)"]
+    }
+  },
+  "grade_comparison": {
+    "A": {
+      "total_score": 95,
+      "grade": "A",
+      "flags": []
+    },
+    "B": {
+      "total_score": 75,
+      "grade": "B",
+      "flags": ["session.list called after task ops", "No admin.help or skill lookup calls"]
+    }
+  },
+  "expectation_results": {
+    "A": {
+      "passed": 5,
+      "total": 5,
+      "pass_rate": 1.0,
+      "details": [
+        {"text": "session.list before any task op", "passed": true},
+        {"text": "session.end called", "passed": true},
+        {"text": "tasks.find used for discovery", "passed": true},
+        {"text": "admin.help called", "passed": true},
+        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
+      ]
+    },
+    "B": {
+      "passed": 3,
+      "total": 5,
+      "pass_rate": 0.60,
+      "details": [
+        {"text": "session.list before any task op", "passed": false},
+        {"text": "session.end called", "passed": true},
+        {"text": "tasks.find used for discovery", "passed": true},
+        {"text": "admin.help called", "passed": false},
+        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
+      ]
+    }
+  }
+}
+```
+
+## Tie Handling
+
+If overall scores are within 0.5 points, declare `"winner": "tie"` and note both performed equivalently.
+
+## Final Summary
+
+After writing comparison.json, output:
+```
+WINNER: <A|B|tie>
+SCORE_A: <overall>
+SCORE_B: <overall>
+GRADE_A: <letter> (<total>/100)
+GRADE_B: <letter> (<total>/100)
+FILE: <comparison.json path>
+```

package/skills/ct-grade/agents/scenario-runner.md

@@ -0,0 +1,134 @@
+# Scenario Runner Agent
+
+You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the specified interface (MCP or CLI), capture the audit trail, and grade the resulting session.
+
+## Inputs
+
+You will receive:
+- `SCENARIO`: Which scenario to run (s1|s2|s3|s4|s5)
+- `INTERFACE`: Which interface to use (mcp|cli)
+- `OUTPUT_DIR`: Where to write results
+- `PROJECT_DIR`: Path to the CLEO project (for cleo-dev)
+- `RUN_NUMBER`: Integer (1, 2, 3...) for repeated runs
+
+## Execution Protocol
+
+### Step 1: Record start time
+
+Note the ISO timestamp before any operations.
+
+### Step 2: Start a graded session via MCP (always use MCP for session lifecycle)
+
+```
+mutate session start { "grade": true, "name": "grade-<SCENARIO>-<INTERFACE>-run<RUN>", "scope": "global" }
+```
+
+Save the returned `sessionId`.
+
+### Step 3: Execute scenario operations
+
+Follow the exact operation sequence from the scenario playbook. Use INTERFACE to determine whether each operation is done via MCP or CLI.
+
+**MCP operations** use the query/mutate gateway:
+```
+query tasks find { "status": "active" }
+```
+
+**CLI operations** use cleo-dev (prefer) or cleo:
+```bash
+cleo-dev find --status active
+```
+
+Scenario sequences are in [../references/scenario-playbook.md](../references/scenario-playbook.md). Execute the operations in order. Do NOT skip operations — each one contributes to the grade.
+
+### Step 4: End the session
+
+```
+mutate session end
+```
+
+### Step 5: Grade the session
+
+```
+query check grade { "sessionId": "<saved-id>" }
+# Compatibility alias: query admin grade { "sessionId": "<saved-id>" }
+```
+
+Save the full GradeResult JSON.
+
+### Step 6: Capture operations log
+
+Record every operation you executed as a JSONL file. Each line:
+```json
+{"seq": 1, "gateway": "query", "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "mcp", "timestamp": "..."}
+```
+
+### Step 7: Write output files
+
+Write to `<OUTPUT_DIR>/<SCENARIO>/arm-<INTERFACE>/`:
+
+**grade.json** — The GradeResult from the canonical `check.grade` read (or legacy `admin.grade` alias):
+```json
+{
+  "sessionId": "...",
+  "totalScore": 85,
+  "maxScore": 100,
+  "dimensions": {...},
+  "flags": [...],
+  "entryCount": 12
+}
+```
+
+**operations.jsonl** — One JSON object per line, each operation executed.
+
+**timing.json** — Fill in what you can; orchestrator fills `total_tokens` and `duration_ms`:
+```json
+{
+  "arm": "<INTERFACE>",
+  "scenario": "<SCENARIO>",
+  "run": <RUN_NUMBER>,
+  "interface": "<INTERFACE>",
+  "executor_start": "<ISO>",
+  "executor_end": "<ISO>",
+  "executor_duration_seconds": 0,
+  "total_tokens": null,
+  "duration_ms": null
+}
+```
+
+Note: `total_tokens` and `duration_ms` are filled by the orchestrator from the task completion notification — you cannot read them yourself.
+
+## Scenario Quick Reference
+
+| Scenario | Key Operations | S1 | S2 | S3 | S4 | S5 |
+|---|---|---|---|---|---|---|
+| s1 | session.list, tasks.find, tasks.show, session.end | ✓ | ✓ | — | — | partial |
+| s2 | session.list, tasks.exists, tasks.add×2, session.end | ✓ | — | ✓ | — | — |
+| s3 | session.list, tasks.show (E_NOT_FOUND), tasks.find (recover), tasks.add, session.end | ✓ | — | ✓ | ✓ | — |
+| s4 | session.list, admin.help, tasks.find, tasks.show, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
+| s5 | session.list, admin.help, tasks.find (parent filter), tasks.show, session.context.drift, session.decision.log, session.record.decision, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
+
+> **S2 scoring note**: The S2 dimension (+5 bonus) requires `tasks.show` to be called after `tasks.find`. Scenarios that only call find but skip show will score 15/20 on S2, not 20/20. Always call tasks.show on at least one result from tasks.find.
+
+## Anti-patterns to Avoid
+
+Do NOT do these during scenario execution — they will lower the grade intentionally only if you are running the anti-pattern variant:
+- Calling `tasks.list` instead of `tasks.find` for discovery
+- Skipping `session.list` at the start
+- Creating tasks without descriptions
+- Ignoring `E_NOT_FOUND` errors without recovery lookup
+- Never calling `admin.help`
+
+## Output
+
+When complete, summarize:
+```
+SCENARIO: <id>
+INTERFACE: <interface>
+RUN: <n>
+SESSION_ID: <id>
+TOTAL_SCORE: <n>/100
+GRADE: <letter>
+FLAGS: <count>
+FILES_WRITTEN: <list>
+```