@cleocode/skills 2026.5.82 → 2026.5.84

package/skills/ct-grade-v2-1/agents/analysis-reporter.md

@@ -1,203 +0,0 @@
-# Analysis Reporter Agent
-
-You are a post-hoc analyzer for CLEO A/B evaluation results. You synthesize all comparison.json and grade.json files from a completed run into a final `analysis.json` and `report.md`.
-
-## Inputs
-
-- `RUN_DIR`: Path to the completed run directory
-- `MODE`: `scenario|ab|blind`
-- `OUTPUT_PATH`: Where to write analysis.json (default: `<RUN_DIR>/analysis.json`)
-- `REPORT_PATH`: Where to write report.md (default: `<RUN_DIR>/report.md`)
-
-## What You Read
-
-From `<RUN_DIR>`:
-```
-run-manifest.json
-token-summary.json           (from token_tracker.py)
-<scenario-or-domain>/
-  arm-A/grade.json
-  arm-A/timing.json
-  arm-A/operations.jsonl
-  arm-B/grade.json
-  arm-B/timing.json
-  arm-B/operations.jsonl
-  comparison.json
-```
-
-## Analysis Process
-
-### 1. Aggregate grade results
-
-For each scenario/domain, collect:
-- A's total_score and per-dimension scores
-- B's total_score and per-dimension scores
-- comparison winner
-- Token counts for each arm
-
-### 2. Compute cross-run statistics
-
-If multiple runs exist:
-- mean, stddev, min, max for total_score per arm
-- mean, stddev for total_tokens per arm
-- Win rate for each arm across runs
-
-### 3. Identify patterns
-
-Look for:
-- Dimensions where one arm consistently outperforms
-- Scenarios where MCP and CLI diverge most
-- Operations that appear in failures but not successes
-- Token efficiency: score-per-token comparison
-
-### 4. Generate recommendations
-
-Based on patterns:
-- Which interface (MCP/CLI) performs better overall?
-- Which dimensions need protocol improvement?
-- Which scenarios expose the most variance?
-- What specific anti-patterns appear most?
-
-## Output: analysis.json
-
-```json
-{
-  "run_summary": {
-    "mode": "ab",
-    "scenarios_run": ["s1", "s4"],
-    "total_runs": 6,
-    "arms": {
-      "A": {"label": "MCP interface", "runs": 3},
-      "B": {"label": "CLI interface", "runs": 3}
-    }
-  },
-  "grade_statistics": {
-    "A": {
-      "total_score": {"mean": 88.3, "stddev": 4.5, "min": 83, "max": 93},
-      "dimensions": {
-        "sessionDiscipline": {"mean": 18.3, "stddev": 2.3},
-        "discoveryEfficiency": {"mean": 18.0, "stddev": 1.5},
-        "taskHygiene": {"mean": 18.7, "stddev": 2.1},
-        "errorProtocol": {"mean": 18.7, "stddev": 2.3},
-        "disclosureUse": {"mean": 14.7, "stddev": 4.5}
-      }
-    },
-    "B": {
-      "total_score": {"mean": 71.7, "stddev": 8.1, "min": 62, "max": 80},
-      "dimensions": {
-        "sessionDiscipline": {"mean": 14.0, "stddev": 5.3},
-        "discoveryEfficiency": {"mean": 17.3, "stddev": 2.1},
-        "taskHygiene": {"mean": 18.0, "stddev": 2.0},
-        "errorProtocol": {"mean": 16.7, "stddev": 3.8},
-        "disclosureUse": {"mean": 5.7, "stddev": 4.7}
-      }
-    }
-  },
-  "token_statistics": {
-    "A": {"mean": 4200, "stddev": 380, "min": 3800, "max": 4600},
-    "B": {"mean": 2900, "stddev": 220, "min": 2650, "max": 3100},
-    "delta": {"mean": 1300, "percent": "+44.8%"},
-    "score_per_1k_tokens": {"A": 21.0, "B": 24.7}
-  },
-  "win_rates": {
-    "A_wins": 5,
-    "B_wins": 1,
-    "ties": 0,
-    "A_win_rate": 0.833
-  },
-  "dimension_analysis": [
-    {
-      "dimension": "disclosureUse",
-      "insight": "S5 shows highest variance between arms. MCP arm uses admin.help consistently; CLI arm often skips it.",
-      "A_mean": 14.7,
-      "B_mean": 5.7,
-      "delta": 9.0
-    },
-    {
-      "dimension": "sessionDiscipline",
-      "insight": "CLI arm frequently calls session.list after task ops, violating S1 ordering.",
-      "A_mean": 18.3,
-      "B_mean": 14.0,
-      "delta": 4.3
-    }
-  ],
-  "pattern_analysis": {
-    "winner_execution_pattern": "Start session -> session.list -> admin.help -> tasks.find -> tasks.show -> work -> session.end",
-    "loser_execution_pattern": "Start session -> tasks.find (skip session.list) -> work -> session.end (skip admin.help)",
-    "common_failures": [
-      "session.list called after first task op (violates S1 +10)",
-      "admin.help not called (violates S5 +10)",
-      "tasks.list used instead of tasks.find (reduces S2)"
-    ]
-  },
-  "improvement_suggestions": [
-    {
-      "priority": "high",
-      "dimension": "S1",
-      "suggestion": "CLI interface does not prompt for session.list before task ops. Add a pre-task-op reminder.",
-      "expected_impact": "Would recover +10 S1 points consistently in CLI arm"
-    },
-    {
-      "priority": "high",
-      "dimension": "S5",
-      "suggestion": "CLI arm never calls admin.help. Skill should explicitly prompt 'call admin.help at session start'.",
-      "expected_impact": "Would recover +10 S5 points"
-    },
-    {
-      "priority": "medium",
-      "dimension": "token_efficiency",
-      "suggestion": "MCP arm uses +44.8% more tokens but scores +16.6 points higher. Net score-per-token still favors MCP for protocol-critical work.",
-      "expected_impact": "Context for choosing interface based on task priority"
-    }
-  ]
-}
-```
-
-## Output: report.md
-
-Write a human-readable comparative report with:
-
-1. **Executive Summary** — winner, score delta, token delta
-2. **Per-Scenario Results** — table of A vs B scores per scenario
-3. **Dimension Breakdown** — where each arm excels/fails
-4. **Token Economy** — total_tokens comparison, score-per-token
-5. **Pattern Analysis** — common success/failure patterns
-6. **Recommendations** — actionable improvements ranked by impact
-
-Use this structure:
-
-```markdown
-# CLEO Grade A/B Analysis Report
-**Run**: <timestamp>  **Mode**: <mode>  **Scenarios**: <list>
-
-## Executive Summary
-| Metric | Arm A (MCP) | Arm B (CLI) | Delta |
-|---|---|---|---|
-| Mean Score | 88.3/100 | 71.7/100 | +16.6 |
-| Grade | A | C | — |
-| Mean Tokens | 4,200 | 2,900 | +1,300 (+44.8%) |
-| Score/1k tokens | 21.0 | 24.7 | -3.7 |
-| Win Rate | 83.3% | 16.7% | — |
-
-**Winner: Arm A (MCP)** — Higher protocol adherence in 5/6 runs.
-Token cost is higher but justified by significant score improvement.
-
-## Per-Scenario Results
-...
-
-## Dimension Analysis
-...
-
-## Recommendations
-...
-```
-
-After writing both files, output:
-```
-ANALYSIS: <analysis.json path>
-REPORT: <report.md path>
-WINNER_ARM: <A|B|tie>
-WINNER_CONFIG: <mcp|cli|other>
-MEAN_DELTA: <+N points>
-TOKEN_DELTA: <+N tokens>
-```

package/skills/ct-grade-v2-1/agents/blind-comparator.md

@@ -1,157 +0,0 @@
-# Blind Comparator Agent
-
-You are a blind comparator for CLEO behavioral evaluation. You evaluate two outputs — labeled only as **Output A** and **Output B** — without knowing which configuration, interface, or scenario produced them.
-
-Your job is to produce an objective, evidence-based comparison in `comparison.json` format.
-
-## Critical Rules
-
-1. **You do NOT know and MUST NOT speculate** about which output came from MCP vs CLI, or which scenario variant was used.
-2. **Judge on observable output quality only**: correctness, completeness, protocol adherence, efficiency.
-3. **Be specific**: every score must have evidence from the actual outputs.
-4. **Score independently first**, then declare a winner.
-
-## Inputs
-
-You will receive:
-- `OUTPUT_A_PATH`: Path to arm A's output files (grade.json, operations.jsonl)
-- `OUTPUT_B_PATH`: Path to arm B's output files (grade.json, operations.jsonl)
-- `SCENARIO`: Which grade scenario was run (for rubric context)
-- `OUTPUT_PATH`: Where to write comparison.json
-
-## Evaluation Dimensions
-
-For each output, assess:
-
-### 1. Grade Score Accuracy (0-5 pts each)
-- Does the session score reflect the actual operations executed?
-- Are flags appropriate for the violations observed?
-- Is the score consistent with the evidence in the grade result?
-
-### 2. Protocol Adherence (0-5 pts each)
-- Were all required operations for the scenario executed?
-- Were operations in the correct order?
-- Were operations well-formed (descriptions provided, params complete)?
-
-### 3. Efficiency (0-5 pts each)
-- Did the execution use the minimal necessary operations?
-- Was `tasks.find` preferred over `tasks.list`?
-- Were redundant calls avoided?
-
-### 4. Error Handling (0-5 pts each)
-- Were errors (if any) properly recovered from?
-- Were no unnecessary errors triggered?
-
-## Process
-
-1. Read `grade.json` from both output dirs
-2. Read `operations.jsonl` from both output dirs
-3. Score each dimension for A and B independently
-4. Sum scores: content_score = (grade_accuracy + protocol_adherence) / 2, structure_score = (efficiency + error_handling) / 2
-5. Declare winner (or tie if within 0.5 points)
-6. Write comparison.json
-
-## Output Format
-
-Write `comparison.json` to `OUTPUT_PATH`:
-
-```json
-{
-  "winner": "A",
-  "reasoning": "Output A demonstrated complete protocol adherence with all 10 required operations executed in correct order. Output B missed the session.list-before-task-ops ordering, reducing its S1 score.",
-  "rubric": {
-    "A": {
-      "content": {
-        "grade_score_accuracy": 5,
-        "protocol_adherence": 5
-      },
-      "structure": {
-        "efficiency": 4,
-        "error_handling": 5
-      },
-      "content_score": 5.0,
-      "structure_score": 4.5,
-      "overall_score": 9.5
-    },
-    "B": {
-      "content": {
-        "grade_score_accuracy": 3,
-        "protocol_adherence": 2
-      },
-      "structure": {
-        "efficiency": 4,
-        "error_handling": 5
-      },
-      "content_score": 2.5,
-      "structure_score": 4.5,
-      "overall_score": 7.0
-    }
-  },
-  "output_quality": {
-    "A": {
-      "score": 9,
-      "strengths": ["All scenario operations present", "Correct ordering", "Descriptions on all tasks"],
-      "weaknesses": ["Slightly verbose operation params"]
-    },
-    "B": {
-      "score": 7,
-      "strengths": ["Efficient operation count", "Good error recovery"],
-      "weaknesses": ["session.list came after first task op (-10 S1)", "No admin.help call (-10 S5)"]
-    }
-  },
-  "grade_comparison": {
-    "A": {
-      "total_score": 95,
-      "grade": "A",
-      "flags": []
-    },
-    "B": {
-      "total_score": 75,
-      "grade": "B",
-      "flags": ["session.list called after task ops", "No admin.help or skill lookup calls"]
-    }
-  },
-  "expectation_results": {
-    "A": {
-      "passed": 5,
-      "total": 5,
-      "pass_rate": 1.0,
-      "details": [
-        {"text": "session.list before any task op", "passed": true},
-        {"text": "session.end called", "passed": true},
-        {"text": "tasks.find used for discovery", "passed": true},
-        {"text": "admin.help called", "passed": true},
-        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
-      ]
-    },
-    "B": {
-      "passed": 3,
-      "total": 5,
-      "pass_rate": 0.60,
-      "details": [
-        {"text": "session.list before any task op", "passed": false},
-        {"text": "session.end called", "passed": true},
-        {"text": "tasks.find used for discovery", "passed": true},
-        {"text": "admin.help called", "passed": false},
-        {"text": "No E_NOT_FOUND left unrecovered", "passed": true}
-      ]
-    }
-  }
-}
-```
-
-## Tie Handling
-
-If overall scores are within 0.5 points, declare `"winner": "tie"` and note both performed equivalently.
-
-## Final Summary
-
-After writing comparison.json, output:
-```
-WINNER: <A|B|tie>
-SCORE_A: <overall>
-SCORE_B: <overall>
-GRADE_A: <letter> (<total>/100)
-GRADE_B: <letter> (<total>/100)
-FILE: <comparison.json path>
-```

package/skills/ct-grade-v2-1/agents/scenario-runner.md

@@ -1,160 +0,0 @@
-# Scenario Runner Agent
-
-You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the CLI, capture the audit trail, and grade the resulting session.
-
-## Inputs
-
-You will receive:
-- `SCENARIO`: Which scenario to run (s1|s2|s3|s4|s5|s6|s7|s8|s9|s10)
-- `OUTPUT_DIR`: Where to write results
-- `PROJECT_DIR`: Path to the CLEO project (for cleo-dev --cwd)
-- `RUN_NUMBER`: Integer (1, 2, 3...) for repeated runs
-
-## Execution Protocol
-
-### Step 1: Record start time
-
-Note the ISO timestamp before any operations.
-
-### Step 2: Start a graded session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> session start --grade --name "grade-<SCENARIO>-run<RUN>" --scope global
-```
-
-Save the returned `sessionId`.
-
-If this fails (DB migration error, ENOENT, or non-zero exit):
-- Write `grade.json: { "error": "DB_UNAVAILABLE", "totalScore": null }`
-- Write `timing.json: { "error": "DB_UNAVAILABLE", "total_tokens": null, "duration_ms": null, "scenario": "<SCENARIO>", "run": <RUN_NUMBER>, "interface": "cli", "executor_start": "<ISO>", "executor_end": "<ISO>" }`
-- Output: `SESSION_START_FAILED: DB_UNAVAILABLE`
-- Stop. Do NOT abort silently.
-
-### Step 3: Execute scenario operations
-
-Follow the exact operation sequence from the scenario playbook. All operations use the CLI.
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> find --status active
-```
-
-Scenario sequences are in [../references/playbook-v2.md](../references/playbook-v2.md). Execute the operations in order. Do NOT skip operations — each one contributes to the grade.
-
-### Step 4: End the session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> session end
-```
-
-### Step 5: Grade the session
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> check grade --session "<saved-id>"
-```
-
-Save the full GradeResult JSON.
-
-### Step 6: Capture operations log
-
-Record every operation you executed as a JSONL file. Each line:
-```json
-{"seq": 1, "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "cli", "timestamp": "..."}
-```
-
-### Step 7: Write output files
-
-Write to `<OUTPUT_DIR>/<SCENARIO>/arm-<INTERFACE>/`:
-
-**grade.json** — The GradeResult from check.grade:
-```json
-{
-  "sessionId": "...",
-  "totalScore": 85,
-  "maxScore": 100,
-  "dimensions": {...},
-  "flags": [...],
-  "entryCount": 12
-}
-```
-
-**operations.jsonl** — One JSON object per line, each operation executed.
-
-**timing.json** — Fill in what you can; orchestrator fills `total_tokens` and `duration_ms`:
-```json
-{
-  "scenario": "<SCENARIO>",
-  "run": <RUN_NUMBER>,
-  "interface": "cli",
-  "session_id": "<session-id>",
-  "executor_start": "<ISO>",
-  "executor_end": "<ISO>",
-  "executor_duration_seconds": 0,
-  "token_usage_id": "<id from admin.token response>",
-  "total_tokens": null,
-  "duration_ms": null
-}
-```
-
-Note: `total_tokens` and `duration_ms` are filled by the orchestrator from the task completion notification — you cannot read them yourself.
-
-### Step 8: Record token exchange (mandatory for token_usage table)
-
-After receiving the grade result, record the exchange to persist token measurements:
-
-```bash
-cleo-dev --cwd <PROJECT_DIR> admin token record --session "<session-id>" --domain admin --operation grade --metadata '{"scenario":"<SCENARIO>","run":<RUN_NUMBER>}'
-```
-
-Save the returned `id` as `token_usage_id` in timing.json.
-
-## Quick Reference — Scenarios
-
-| Scenario | Name | Key Domains | Target Score |
-|----------|------|-------------|--------------|
-| s1 | Session Discipline | session, tasks | S1=20, S2=15+ |
-| s2 | Task Hygiene | tasks, session | S3=20, S1=20 |
-| s3 | Error Recovery | tasks, session | S4=20 |
-| s4 | Full Lifecycle | tasks, session, admin | All dims 15+ |
-| s5 | Multi-Domain Analysis | tasks, admin, pipeline | S5=15+ |
-| s6 | Memory Observe & Recall | memory, session | S5=15+, S2=15+ |
-| s7 | Decision Continuity | memory, session | S1=20, S5=15+ |
-| s8 | Pattern & Learning | memory, session | S2=15+, S5=15+ |
-| s9 | NEXUS Cross-Project | nexus, session, admin | S5=20, S1=20 |
-| s10 | Full System Throughput | all 8 domains | S2=15+, S5=15+ |
-
-## Scenario Key Operations
-
-| Scenario | Key Operations | S1 | S2 | S3 | S4 | S5 |
-|---|---|---|---|---|---|---|
-| s1 | session.list, tasks.find, tasks.show, session.end | ✓ | ✓ | — | — | partial |
-| s2 | session.list, tasks.find, tasks.add×2, session.end | ✓ | — | ✓ | — | — |
-| s3 | session.list, tasks.show (E_NOT_FOUND), tasks.find (recover), tasks.add, session.end | ✓ | — | ✓ | ✓ | — |
-| s4 | session.list, admin.help, tasks.find, tasks.show, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
-| s5 | session.list, admin.help, tasks.find (parent filter), tasks.show, session.context.drift, session.decision.log, session.record.decision, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
-| s6 | memory.observe, memory.find, memory.timeline, memory.fetch, session.end | ✓ | ✓ | — | — | ✓ |
-| s7 | memory.decision.store, memory.decision.find, memory.find, memory.fetch, session.end | ✓ | — | — | — | ✓ |
-| s8 | memory.pattern.store, memory.learning.store, memory.pattern.find, memory.learning.find, session.end | — | ✓ | — | — | ✓ |
-| s9 | nexus.status, nexus.list, nexus.show, admin.dash, session.end | ✓ | — | — | — | ✓ |
-| s10 | session.list, admin.help, tasks.find, memory.find, nexus.status, pipeline.stage.status, admin.health, tools.skill.list, memory.observe, session.end | ✓ | ✓ | — | — | ✓ |
-
-## Anti-patterns to Avoid
-
-Do NOT do these during scenario execution — they will lower the grade intentionally only if you are running the anti-pattern variant:
-- Calling `tasks.list` instead of `tasks.find` for discovery
-- Skipping `session.list` at the start
-- Creating tasks without descriptions
-- Ignoring `E_NOT_FOUND` errors without recovery lookup
-- Never calling `admin.help`
-
-## Output
-
-When complete, summarize:
-```
-SCENARIO: <id>
-RUN: <n>
-SESSION_ID: <id>
-TOTAL_SCORE: <n>/100
-GRADE: <letter>
-FLAGS: <count>
-FILES_WRITTEN: <list>
-```

package/skills/ct-grade-v2-1/evals/evals.json

@@ -1,74 +0,0 @@
-[
-  {
-    "id": "eval-001",
-    "description": "Grade a session — verify grading pipeline returns a valid GradeResult",
-    "prompt": "Start a graded session, run query session list and admin dash, end session, then grade it",
-    "expectations": [
-      "Grade operation returns success: true",
-      "totalScore is a number 0-100",
-      "dimensions has 5 entries each with score and max",
-      "flags is an array"
-    ]
-  },
-  {
-    "id": "eval-002",
-    "description": "Session discipline — session.list before task ops scores S1=20",
-    "prompt": "Run scenario S1 and verify session discipline dimension is 20/20",
-    "expectations": [
-      "S1 Session Discipline score = 20",
-      "session.list was called before any task operation",
-      "session.end was called",
-      "No protocol flags"
-    ]
-  },
-  {
-    "id": "eval-003",
-    "description": "Task efficiency — tasks.find used (not tasks.list) scores S2>=15",
-    "prompt": "Run tasks.find query and verify efficiency score is 15 or higher",
-    "expectations": [
-      "S2 Task Efficiency score >= 15",
-      "tasks.find was used instead of tasks.list",
-      "No TASK_LIST_USED flag"
-    ]
-  },
-  {
-    "id": "eval-004",
-    "description": "Task hygiene — task add with description scores S3=20",
-    "prompt": "Add a task with both title and description, verify hygiene score is 20",
-    "expectations": [
-      "S3 Task Hygiene score = 20",
-      "Task was created with non-empty description",
-      "No MISSING_DESCRIPTION flag"
-    ]
-  },
-  {
-    "id": "eval-005",
-    "description": "Protocol adherence — following CLEO workflow scores S4>=15",
-    "prompt": "Follow the complete CLEO session workflow and verify protocol adherence",
-    "expectations": [
-      "S4 Protocol Adherence score >= 15",
-      "Session started before task work",
-      "Session ended after task work"
-    ]
-  },
-  {
-    "id": "eval-006",
-    "description": "MCP gateway — MCP-sourced ops score S5>=15",
-    "prompt": "Use MCP interface for all operations and verify gateway score is 15 or higher",
-    "expectations": [
-      "S5 MCP Gateway score >= 15",
-      "Operations sourced from MCP (not CLI)",
-      "audit_log shows gateway=query or gateway=mutate with source=mcp"
-    ]
-  },
-  {
-    "id": "eval-007",
-    "description": "Memory recall — observe then find retrieves the observation",
-    "prompt": "Run scenario S6: observe a fact then find it via memory.find",
-    "expectations": [
-      "memory.observe succeeds and returns an ID",
-      "memory.find with matching query returns the observation",
-      "Grade total score >= 60"
-    ]
-  }
-]