@cleocode/skills 2.0.0

package/skills/ct-grade-v2-1/agents/scenario-runner.md

@@ -0,0 +1,179 @@
+# Scenario Runner Agent
+
+You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the specified interface (MCP or CLI), capture the audit trail, and grade the resulting session.
+
+## Inputs
+
+You will receive:
+- `SCENARIO`: Which scenario to run (s1|s2|s3|s4|s5|s6|s7|s8|s9|s10)
+- `INTERFACE`: Which interface to use (mcp|cli)
+- `OUTPUT_DIR`: Where to write results
+- `PROJECT_DIR`: Path to the CLEO project (for cleo-dev --cwd)
+- `RUN_NUMBER`: Integer (1, 2, 3...) for repeated runs
+
+## Execution Protocol
+
+### Step 1: Record start time
+
+Note the ISO timestamp before any operations.
+
+### Step 2: Start a graded session via MCP (always use MCP for session lifecycle)
+
+```
+mutate session start { "grade": true, "name": "grade-<SCENARIO>-<INTERFACE>-run<RUN>", "scope": "global" }
+```
+
+Save the returned `sessionId`.
+
+If this fails (DB migration error, ENOENT, or non-zero exit):
+- Write `grade.json: { "error": "DB_UNAVAILABLE", "totalScore": null }`
+- Write `timing.json: { "error": "DB_UNAVAILABLE", "total_tokens": null, "duration_ms": null, "arm": "<INTERFACE>", "scenario": "<SCENARIO>", "run": <RUN_NUMBER>, "interface": "<INTERFACE>", "executor_start": "<ISO>", "executor_end": "<ISO>" }`
+- Output: `SESSION_START_FAILED: DB_UNAVAILABLE`
+- Stop. Do NOT abort silently.
+
+### Step 3: Execute scenario operations
+
+Follow the exact operation sequence from the scenario playbook. Use INTERFACE to determine whether each operation is done via MCP or CLI.
+
+**MCP operations** use the query/mutate gateway:
+```
+query tasks find { "status": "active" }
+```
+
+**CLI operations** use cleo-dev (prefer) or cleo, with PROJECT_DIR as cwd if provided:
+```bash
+cleo-dev --cwd <PROJECT_DIR> find --status active
+```
+
+Scenario sequences are in [../references/playbook-v2.md](../references/playbook-v2.md). Execute the operations in order. Do NOT skip operations — each one contributes to the grade.
+
+### Step 4: End the session
+
+```
+mutate session end
+```
+
+### Step 5: Grade the session
+
+```
+query admin grade { "sessionId": "<saved-id>" }
+```
+
+Save the full GradeResult JSON.
+
+### Step 6: Capture operations log
+
+Record every operation you executed as a JSONL file. Each line:
+```json
+{"seq": 1, "gateway": "query", "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "mcp", "timestamp": "..."}
+```
+
+### Step 7: Write output files
+
+Write to `<OUTPUT_DIR>/<SCENARIO>/arm-<INTERFACE>/`:
+
+**grade.json** — The GradeResult from admin.grade:
+```json
+{
+  "sessionId": "...",
+  "totalScore": 85,
+  "maxScore": 100,
+  "dimensions": {...},
+  "flags": [...],
+  "entryCount": 12
+}
+```
+
+**operations.jsonl** — One JSON object per line, each operation executed.
+
+**timing.json** — Fill in what you can; orchestrator fills `total_tokens` and `duration_ms`:
+```json
+{
+  "arm": "<INTERFACE>",
+  "scenario": "<SCENARIO>",
+  "run": <RUN_NUMBER>,
+  "interface": "<INTERFACE>",
+  "session_id": "<session-id>",
+  "executor_start": "<ISO>",
+  "executor_end": "<ISO>",
+  "executor_duration_seconds": 0,
+  "token_usage_id": "<id from admin.token.record response>",
+  "total_tokens": null,
+  "duration_ms": null
+}
+```
+
+Note: `total_tokens` and `duration_ms` are filled by the orchestrator from the task completion notification — you cannot read them yourself.
+
+### Step 8: Record token exchange (mandatory for token_usage table)
+
+After receiving the grade result, record the exchange to persist token measurements:
+
+```
+mutate admin token.record {
+  "sessionId": "<session-id>",
+  "transport": "mcp",
+  "domain": "admin",
+  "operation": "grade",
+  "metadata": {
+    "scenario": "<SCENARIO>",
+    "interface": "<INTERFACE>",
+    "run": <RUN_NUMBER>
+  }
+}
+```
+
+Save the returned `id` as `token_usage_id` in timing.json.
+
+## Quick Reference — Scenarios
+
+| Scenario | Name | Key Domains | Target Score |
+|----------|------|-------------|--------------|
+| s1 | Session Discipline | session, tasks | S1=20, S2=15+ |
+| s2 | Task Hygiene | tasks, session | S3=20, S1=20 |
+| s3 | Error Recovery | tasks, session | S4=20 |
+| s4 | Full Lifecycle | tasks, session, admin | All dims 15+ |
+| s5 | Multi-Domain Analysis | tasks, admin, pipeline | S5=15+ |
+| s6 | Memory Observe & Recall | memory, session | S5=15+, S2=15+ |
+| s7 | Decision Continuity | memory, session | S1=20, S5=15+ |
+| s8 | Pattern & Learning | memory, session | S2=15+, S5=15+ |
+| s9 | NEXUS Cross-Project | nexus, session, admin | S5=20, S1=20 |
+| s10 | Full System Throughput | all 8 domains | S2=15+, S5=15+ |
+
+## Scenario Key Operations
+
+| Scenario | Key Operations | S1 | S2 | S3 | S4 | S5 |
+|---|---|---|---|---|---|---|
+| s1 | session.list, tasks.find, tasks.show, session.end | ✓ | ✓ | — | — | partial |
+| s2 | session.list, tasks.exists, tasks.add×2, session.end | ✓ | — | ✓ | — | — |
+| s3 | session.list, tasks.show (E_NOT_FOUND), tasks.find (recover), tasks.add, session.end | ✓ | — | ✓ | ✓ | — |
+| s4 | session.list, admin.help, tasks.find, tasks.show, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
+| s5 | session.list, admin.help, tasks.find (parent filter), tasks.show, session.context.drift, session.decision.log, session.record.decision, tasks.update, tasks.complete, session.end | ✓ | ✓ | ✓ | ✓ | ✓ |
+| s6 | memory.observe, memory.find, memory.timeline, memory.fetch, session.end | ✓ | ✓ | — | — | ✓ |
+| s7 | memory.decision.store, memory.decision.find, memory.find, memory.stats, session.end | ✓ | — | — | — | ✓ |
+| s8 | memory.pattern.store, memory.learning.store, memory.pattern.find, memory.learning.find, session.end | — | ✓ | — | — | ✓ |
+| s9 | nexus.status, nexus.list, nexus.show, admin.dash, session.end | ✓ | — | — | — | ✓ |
+| s10 | session.list, admin.help, tasks.find, memory.find, nexus.status, pipeline.stage.status, check.health, tools.skill.list, memory.observe, session.end | ✓ | ✓ | — | — | ✓ |
+
+## Anti-patterns to Avoid
+
+Do NOT do these during scenario execution — they will lower the grade intentionally only if you are running the anti-pattern variant:
+- Calling `tasks.list` instead of `tasks.find` for discovery
+- Skipping `session.list` at the start
+- Creating tasks without descriptions
+- Ignoring `E_NOT_FOUND` errors without recovery lookup
+- Never calling `admin.help`
+
+## Output
+
+When complete, summarize:
+```
+SCENARIO: <id>
+INTERFACE: <interface>
+RUN: <n>
+SESSION_ID: <id>
+TOTAL_SCORE: <n>/100
+GRADE: <letter>
+FLAGS: <count>
+FILES_WRITTEN: <list>
+```

package/skills/ct-grade-v2-1/evals/evals.json

@@ -0,0 +1,74 @@
+[
+  {
+    "id": "eval-001",
+    "description": "Grade a session — verify grading pipeline returns a valid GradeResult",
+    "prompt": "Start a graded session, run query session list and admin dash, end session, then grade it",
+    "expectations": [
+      "Grade operation returns success: true",
+      "totalScore is a number 0-100",
+      "dimensions has 5 entries each with score and max",
+      "flags is an array"
+    ]
+  },
+  {
+    "id": "eval-002",
+    "description": "Session discipline — session.list before task ops scores S1=20",
+    "prompt": "Run scenario S1 and verify session discipline dimension is 20/20",
+    "expectations": [
+      "S1 Session Discipline score = 20",
+      "session.list was called before any task operation",
+      "session.end was called",
+      "No protocol flags"
+    ]
+  },
+  {
+    "id": "eval-003",
+    "description": "Task efficiency — tasks.find used (not tasks.list) scores S2>=15",
+    "prompt": "Run tasks.find query and verify efficiency score is 15 or higher",
+    "expectations": [
+      "S2 Task Efficiency score >= 15",
+      "tasks.find was used instead of tasks.list",
+      "No TASK_LIST_USED flag"
+    ]
+  },
+  {
+    "id": "eval-004",
+    "description": "Task hygiene — task add with description scores S3=20",
+    "prompt": "Add a task with both title and description, verify hygiene score is 20",
+    "expectations": [
+      "S3 Task Hygiene score = 20",
+      "Task was created with non-empty description",
+      "No MISSING_DESCRIPTION flag"
+    ]
+  },
+  {
+    "id": "eval-005",
+    "description": "Protocol adherence — following CLEO workflow scores S4>=15",
+    "prompt": "Follow the complete CLEO session workflow and verify protocol adherence",
+    "expectations": [
+      "S4 Protocol Adherence score >= 15",
+      "Session started before task work",
+      "Session ended after task work"
+    ]
+  },
+  {
+    "id": "eval-006",
+    "description": "MCP gateway — MCP-sourced ops score S5>=15",
+    "prompt": "Use MCP interface for all operations and verify gateway score is 15 or higher",
+    "expectations": [
+      "S5 MCP Gateway score >= 15",
+      "Operations sourced from MCP (not CLI)",
+      "audit_log shows gateway=query or gateway=mutate with source=mcp"
+    ]
+  },
+  {
+    "id": "eval-007",
+    "description": "Memory recall — observe then find retrieves the observation",
+    "prompt": "Run scenario S6: observe a fact then find it via memory.find",
+    "expectations": [
+      "memory.observe succeeds and returns an ID",
+      "memory.find with matching query returns the observation",
+      "Grade total score >= 60"
+    ]
+  }
+]

package/skills/ct-grade-v2-1/grade-viewer/build_op_stats.py

@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""
+build_op_stats.py — Aggregate operations.jsonl files from grade runs into per-operation statistics.
+
+Reads all operations.jsonl files under --grade-runs-dir and computes per-operation stats
+split by interface (mcp/cli). Output is a JSON object keyed by "domain.operation".
+
+Usage:
+    python build_op_stats.py [options]
+
+Options:
+    --grade-runs-dir PATH    Directory containing grade run subdirectories
+                             (default: .cleo/metrics/grade-runs relative to cwd)
+    --output PATH            Output JSON file path
+                             (default: .cleo/metrics/per_operation_stats.json)
+    --pretty                 Pretty-print JSON output (default: compact)
+    --verbose                Print progress to stderr
+
+Output format (per key "domain.operation"):
+{
+    "mcp_calls": 42,
+    "cli_calls": 10,
+    "total_mcp_ms": 1234.5,
+    "total_cli_ms": 456.7,
+    "avg_mcp_ms": 29.4,
+    "avg_cli_ms": 45.7,
+    "runs_seen": 3
+}
+
+Also importable as a module:
+    from build_op_stats import compute_stats
+    stats = compute_stats(grade_runs_dir="/path/to/grade-runs")
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+
+def compute_stats(grade_runs_dir, verbose=False):
+    """
+    Aggregate operations.jsonl files under grade_runs_dir.
+
+    Returns dict keyed by "domain.operation" with accumulated stats.
+    """
+    runs_dir = Path(grade_runs_dir)
+    stats = {}
+    files_processed = 0
+    lines_processed = 0
+
+    if not runs_dir.exists():
+        if verbose:
+            print(f"[build_op_stats] Grade runs dir not found: {runs_dir}", file=sys.stderr)
+        return stats
+
+    for ops_file in sorted(runs_dir.rglob('operations.jsonl')):
+        files_processed += 1
+        if verbose:
+            print(f"[build_op_stats] Processing: {ops_file}", file=sys.stderr)
+
+        for line in ops_file.read_text(errors='replace').splitlines():
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                entry = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+
+            domain = entry.get('domain', 'unknown')
+            operation = entry.get('operation', 'unknown')
+            key = f"{domain}.{operation}"
+            interface = entry.get('interface', 'mcp')
+            duration = float(entry.get('duration_ms', 0) or 0)
+
+            if key not in stats:
+                stats[key] = {
+                    'mcp_calls': 0,
+                    'cli_calls': 0,
+                    'total_mcp_ms': 0.0,
+                    'total_cli_ms': 0.0,
+                    'avg_mcp_ms': 0.0,
+                    'avg_cli_ms': 0.0,
+                    'runs_seen': set(),
+                }
+
+            # Track which run directory this came from
+            # ops_file is e.g. .../grade-runs/run-20260308/s1/run-01/arm-mcp/operations.jsonl
+            # run_id is the first path component relative to runs_dir (e.g. "run-20260308")
+            run_id = ops_file.relative_to(runs_dir).parts[0]
+            stats[key]['runs_seen'].add(run_id)
+
+            if interface == 'cli':
+                stats[key]['cli_calls'] += 1
+                stats[key]['total_cli_ms'] += duration
+            else:
+                stats[key]['mcp_calls'] += 1
+                stats[key]['total_mcp_ms'] += duration
+
+            lines_processed += 1
+
+    # Compute averages and convert sets to counts
+    for key, v in stats.items():
+        mc = v['mcp_calls']
+        cc = v['cli_calls']
+        v['avg_mcp_ms'] = round(v['total_mcp_ms'] / mc, 2) if mc > 0 else 0.0
+        v['avg_cli_ms'] = round(v['total_cli_ms'] / cc, 2) if cc > 0 else 0.0
+        v['total_mcp_ms'] = round(v['total_mcp_ms'], 2)
+        v['total_cli_ms'] = round(v['total_cli_ms'], 2)
+        v['runs_seen'] = len(v['runs_seen'])
+
+    if verbose:
+        print(f"[build_op_stats] Processed {files_processed} files, {lines_processed} lines → {len(stats)} unique operations", file=sys.stderr)
+
+    return stats
+
+
+def find_cleo_dir(start='.'):
+    """Walk up from start to find directory containing .cleo/tasks.db."""
+    p = Path(start).resolve()
+    while p != p.parent:
+        if (p / '.cleo' / 'tasks.db').exists():
+            return p
+        p = p.parent
+    return Path(start).resolve()
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description='Aggregate grade run operations.jsonl files into per-operation stats.'
+    )
+    parser.add_argument(
+        '--grade-runs-dir',
+        default=None,
+        help='Directory containing grade run subdirectories (default: .cleo/metrics/grade-runs)'
+    )
+    parser.add_argument(
+        '--output',
+        default=None,
+        help='Output JSON path (default: .cleo/metrics/per_operation_stats.json)'
+    )
+    parser.add_argument(
+        '--pretty',
+        action='store_true',
+        help='Pretty-print JSON output'
+    )
+    parser.add_argument(
+        '--verbose',
+        action='store_true',
+        help='Print progress to stderr'
+    )
+    args = parser.parse_args()
+
+    workspace = find_cleo_dir('.')
+
+    grade_runs_dir = args.grade_runs_dir or str(workspace / '.cleo' / 'metrics' / 'grade-runs')
+    output_path = args.output or str(workspace / '.cleo' / 'metrics' / 'per_operation_stats.json')
+
+    stats = compute_stats(grade_runs_dir, verbose=args.verbose)
+
+    indent = 2 if args.pretty else None
+    output_json = json.dumps(stats, indent=indent)
+
+    out = Path(output_path)
+    out.parent.mkdir(parents=True, exist_ok=True)
+    out.write_text(output_json)
+
+    print(f"Wrote {len(stats)} operation stats to {output_path}")
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

package/skills/ct-grade-v2-1/grade-viewer/eval-analysis.json

@@ -0,0 +1,41 @@
+{
+  "total_grades": 31,
+  "score_distribution": {
+    "F (0)": 7,
+    "D (45-59)": 10,
+    "C (60-74)": 5,
+    "B (75-89)": 8,
+    "A (90+)": 1
+  },
+  "score_stats": {
+    "mean": 64.6,
+    "min": 50,
+    "max": 95,
+    "grades_with_data": 24,
+    "zero_score_count": 7
+  },
+  "dimension_averages": {
+    "sessionDiscipline": 5.8,
+    "discoveryEfficiency": 9.0,
+    "taskHygiene": 15.4,
+    "errorProtocol": 15.3,
+    "disclosureUse": 4.5
+  },
+  "flag_frequency": {
+    "No admin.help calls": 21,
+    "session.list never called": 18,
+    "No MCP query calls": 13,
+    "session.end never called": 12,
+    "No audit entries": 7,
+    "tasks.list used (prefer find)": 5,
+    "Subtasks without exists check": 1,
+    "Duplicate task creates": 1
+  },
+  "avg_audit_entries": 9.5,
+  "token_estimate": {
+    "avg_per_session_chars": 0,
+    "avg_per_session_tokens": 1425.0,
+    "method": "entry_count * 150 proxy",
+    "note": "OTEL not enabled; enable with CLAUDE_CODE_ENABLE_TELEMETRY=1 for real counts"
+  }
+}

package/skills/ct-grade-v2-1/grade-viewer/eval-report.md

@@ -0,0 +1,34 @@
+# CLEO Grade v2.1 — Comparative Analysis Report
+
+**Generated:** 2026-03-07 23:47 UTC  
+**Source:** `/tmp/ct-grade-eval`
+
+---
+
+## MCP vs CLI Blind A/B Results
+
+**Overall winner: MCP**  
+
+| Metric | Value |
+|--------|-------|
+| Total runs | 3 |
+| MCP wins | 3 (100.0%) |
+| CLI wins | 0 (0.0%) |
+| Ties | 0 |
+| Avg token delta (MCP–CLI) | +416.0 tokens |
+| Interpretation | MCP uses more tokens on average |
+
+### Per-Operation Results
+
+| Operation | MCP wins | CLI wins | Ties | Token delta | MCP chars | CLI chars | MCP ms | CLI ms |
+|-----------|----------|----------|------|-------------|-----------|-----------|--------|--------|
+| `admin.version` **MCP** | 3 | 0 | 0 | +416t | 1664 | 0 | 930ms | 786ms |
+
+### Recommendations
+
+- **MCP adds significant token overhead.** Consider whether MCP envelope verbosity can be reduced for high-frequency operations.
+- **MCP output quality is consistently higher.** Reinforces MCP-first agent protocol recommendation.
+
+---
+
+*Report generated by ct-grade v2.1 `generate_report.py`*