@cleocode/skills 2026.3.76 → 2026.4.2

package/skills/ct-grade-v2-1/references/ab-testing.md

@@ -1,18 +1,21 @@
 # Blind A/B Testing Protocol
 
-Methodology for blind comparison of MCP vs CLI interface usage in CLEO.
+Methodology for blind comparison of grade scenario results in CLEO.
+
+> **Note**: MCP support was removed. All operations now use the CLI exclusively.
+> This protocol compares different CLI configurations, binary versions, or parameter sets.
 
 ---
 
 ## Agent-Based Execution (Canonical)
 
-The canonical A/B approach uses Claude Code Agents to run scenarios end-to-end via the live MCP/CLI interfaces. This avoids subprocess initialization issues and captures real token data from task notifications.
+The canonical A/B approach uses Claude Code Agents to run scenarios end-to-end via the CLI. This captures real token data from task notifications.
 
 ### Execution Flow
 
 1. Run `python scripts/setup_run.py` to create run structure and print the execution plan
-2. Follow the plan: spawn scenario-runner agents in parallel (arm-A MCP, arm-B CLI)
-3. Immediately capture `total_tokens` from each task notification → `timing.json`
+2. Follow the plan: spawn scenario-runner agents in parallel (arm-A, arm-B with different configurations)
+3. Immediately capture `total_tokens` from each task notification to `timing.json`
 4. Spawn blind-comparator agent after both arms complete
 5. Run `python scripts/token_tracker.py --run-dir <dir>` to aggregate tokens
 6. Run `python scripts/generate_report.py --run-dir <dir>` for final report
@@ -22,10 +25,10 @@ The canonical A/B approach uses Claude Code Agents to run scenarios end-to-end v
 ```python
 # After EACH agent task completes, fill timing.json immediately:
 timing = {
-  "total_tokens": task.total_tokens,   # EPHEMERAL — capture now or lose it
+  "total_tokens": task.total_tokens,   # EPHEMERAL -- capture now or lose it
   "duration_ms": task.duration_ms,
   "arm": "arm-A",
-  "interface": "mcp",
+  "interface": "cli",
   "scenario": "s4",
   "run": 1,
 }
@@ -35,30 +38,7 @@ Token data priority:
 1. `total_tokens` from Claude Code Agent task notification (canonical)
 2. OTel `claude_code.token.usage` (when `CLAUDE_CODE_ENABLE_TELEMETRY=1`)
 3. `output_chars / 3.5` (JSON response estimate)
-4. `entryCount × 150` (coarse proxy from GRADES.jsonl)
-
----
-
-## Subprocess-Based Execution (Fallback)
-
-For automated testing without agent delegation, use `run_ab_test.py`. This invokes CLEO via subprocess and requires a migrated `tasks.db`.
-
----
-
-## What We're Testing
-
-| Side | Interface | Mechanism |
-|------|-----------|-----------|
-| **A** (MCP) | JSON-RPC via stdio to CLEO MCP server | `node dist/mcp/index.js` with JSON-RPC messages |
-| **B** (CLI) | Shell commands via subprocess | `cleo-dev <domain> <operation> [params]` |
-
-Both sides call the same underlying `src/dispatch/` layer. The A/B test isolates:
-- **Output format differences** — MCP returns structured JSON envelopes; CLI may add ANSI/formatting
-- **Response size** — character counts as token proxy
-- **Latency** — wall-clock time per operation
-- **Data equivalence** — do they return the same logical data?
-
-Blind assignment means the comparator does not know which result came from MCP vs CLI when producing the quality verdict.
+4. `entryCount x 150` (coarse proxy from GRADES.jsonl)
 
 ---
 
@@ -89,11 +69,11 @@ ab-results/
 ## Blind Assignment
 
 The `run_ab_test.py` script randomly shuffles which side gets labeled "A" vs "B" for each run. The comparator agent sees only:
-- Output labeled "A" (could be MCP or CLI)
-- Output labeled "B" (could be MCP or CLI)
+- Output labeled "A"
+- Output labeled "B"
 - The original request prompt
 
-The `meta.json` records the true identity (`a_is_mcp: true|false`) per run. `generate_report.py` de-blinds after all comparisons are done.
+The `meta.json` records the true identity per run. `generate_report.py` de-blinds after all comparisons are done.
 
 ---
 
@@ -104,7 +84,7 @@ The `meta.json` records the true identity (`a_is_mcp: true|false`) per run. `gen
 | `output_chars` | `len(response_json_str)` |
 | `estimated_tokens` | `output_chars / 4` (approximation) |
 | `duration_ms` | wall clock from subprocess start to end |
-| `success` | `response.success === true` (MCP) or exit code 0 (CLI) |
+| `success` | exit code 0 |
 | `data_equivalent` | compare key fields between A and B response |
 
 ---
@@ -136,17 +116,17 @@ After N runs, `generate_report.py` computes:
 
 ```json
 {
-  "wins": { "mcp": 0, "cli": 0, "tie": 0 },
-  "win_rate": { "mcp": 0.0, "cli": 0.0 },
+  "wins": { "arm_a": 0, "arm_b": 0, "tie": 0 },
+  "win_rate": { "arm_a": 0.0, "arm_b": 0.0 },
   "token_delta": {
-    "mean_mcp_chars": 0,
-    "mean_cli_chars": 0,
+    "mean_a_chars": 0,
+    "mean_b_chars": 0,
     "delta_chars": 0,
     "delta_pct": "+0%"
   },
   "latency_delta": {
-    "mean_mcp_ms": 0,
-    "mean_cli_ms": 0,
+    "mean_a_ms": 0,
+    "mean_b_ms": 0,
     "delta_ms": 0
   },
   "data_equivalence_rate": 1.0,
@@ -167,48 +147,18 @@ The blind comparator evaluates each side on:
 | **Completeness** | Does the response contain all expected fields? |
 | **Structure** | Is the response well-formed JSON? Clean envelope? |
 | **Usability** | Can an agent consume this without post-processing? |
-| **Verbosity** | Lower is better — same data, fewer chars = more efficient |
+| **Verbosity** | Lower is better -- same data, fewer chars = more efficient |
 
-Rubric scores are 1–5 per criterion. Winner is the side with higher weighted total.
+Rubric scores are 1-5 per criterion. Winner is the side with higher weighted total.
 
 ---
 
-## MCP Server Invocation Details
+## CLI Invocation
 
-The `run_ab_test.py` script calls the CLEO MCP server via stdio JSON-RPC:
+All operations use the CLI:
 
-```python
-# Protocol sequence
-# 1. Send initialize
-# 2. Send tools/call (query or mutate)
-# 3. Read response lines until tool result found
-# 4. Terminate process
-
-MCP_INIT = {
-  "jsonrpc": "2.0", "id": 0, "method": "initialize",
-  "params": {
-    "protocolVersion": "2024-11-05",
-    "capabilities": {},
-    "clientInfo": {"name": "ct-grade-ab-test", "version": "2.1.0"}
-  }
-}
-
-MCP_CALL = {
-  "jsonrpc": "2.0", "id": 1, "method": "tools/call",
-  "params": {
-    "name": "query",  # or "mutate"
-    "arguments": {
-      "domain": "<domain>",
-      "operation": "<operation>",
-      "params": {}
-    }
-  }
-}
-```
-
-**CLI equivalent:**
 ```bash
-cleo-dev <domain> <operation> [args] --json
+cleo-dev <command> [args] --json
 ```
 
 ---
@@ -217,17 +167,7 @@ cleo-dev <domain> <operation> [args] --json
 
 | Outcome | Meaning | Action |
 |---------|---------|--------|
-| MCP wins consistently | MCP output is cleaner/more complete | Recommend MCP-first in agent protocols |
-| CLI wins consistently | CLI output is more complete or parseable | Investigate MCP envelope overhead |
+| Arm A wins consistently | Configuration A output is cleaner/more complete | Investigate differences |
+| Arm B wins consistently | Configuration B output is more complete or parseable | Investigate differences |
 | Tie | Both equivalent | Focus on latency and token cost |
-| MCP tokens > CLI tokens | MCP envelope adds overhead | Quantify and document in CLEO-GRADE-SPEC |
-| Data divergence detected | MCP and CLI returning different data | File bug — should be dispatch-level consistent |
-
----
-
-## Parity Scenarios
-
-The P1-P3 parity scenarios (see playbook-v2.md) run a curated set of operations specifically chosen to stress:
-- **P1**: tasks domain — high-frequency agent operations
-- **P2**: session domain — lifecycle operations agents use at start/end
-- **P3**: admin domain — help, dash, health (first calls in any session)
+| Data divergence detected | Arms returning different data | File bug -- should be consistent |

package/skills/ct-grade-v2-1/references/grade-spec-v2.md

@@ -82,16 +82,16 @@ Measures whether the agent recovers from `E_NOT_FOUND` (exit code 4) and avoids
 
 ### S5: Progressive Disclosure Use (20 pts)
 
-Measures whether the agent uses CLEO's progressive disclosure system and the MCP query gateway.
+Measures whether the agent uses CLEO's progressive disclosure system.
 
 | Points | Condition | Evidence string |
 |--------|-----------|-----------------|
 | +10 | At least one help/skill call: `admin.help`, `tools.skill.show`, `tools.skill.list`, `tools.skill.find` | `Progressive disclosure used (Nx)` |
-| +10 | At least one MCP query gateway call (`metadata.gateway === "query"`) | `query (MCP) used Nx` |
+| +10 | Progressive disclosure used for efficient access | `Progressive disclosure active` |
 
 **Flags:**
 - `No admin.help or skill lookup calls (load ct-cleo for guidance)`
-- `No MCP query calls (prefer query over CLI for programmatic access)`
+- `No progressive disclosure calls (use admin.help or skill lookups)`
 
 **Scoring:** Starts at 0. Range: 0–20.
 
@@ -123,7 +123,7 @@ Grade results in v2.1 carry optional token metadata alongside the standard Grade
       "session": 600,
       "admin": 400
     },
-    "mcpQueryTokens": 2100,
+    "queryTokens": 2100,
     "cliTokens": 1100,
     "auditEntries": 47
   }
@@ -164,4 +164,4 @@ The rubric recognizes all 10 canonical domains in audit entries. Key domain-to-d
 | `nexus` | S5 (gateway tracking only) |
 | `sticky` | S5 (gateway tracking only) |
 
-All 10 domains contribute to `mcpQueryCalls` count in S5 — any MCP query gateway call regardless of domain earns the +10.
+All 10 domains contribute to the progressive disclosure count in S5 — any help or skill lookup call regardless of domain earns the +10.