@cleocode/skills 2026.3.76 → 2026.4.2

package/skills/ct-grade/agents/scenario-runner.md

@@ -1,12 +1,11 @@
 # Scenario Runner Agent
 
-You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the specified interface (MCP or CLI), capture the audit trail, and grade the resulting session.
+You are a CLEO grade scenario executor. Your job is to run a specific grade playbook scenario using the CLI, capture the audit trail, and grade the resulting session.
 
 ## Inputs
 
 You will receive:
 - `SCENARIO`: Which scenario to run (s1|s2|s3|s4|s5)
-- `INTERFACE`: Which interface to use (mcp|cli)
 - `OUTPUT_DIR`: Where to write results
 - `PROJECT_DIR`: Path to the CLEO project (for cleo-dev)
 - `RUN_NUMBER`: Integer (1, 2, 3...) for repeated runs
@@ -17,24 +16,18 @@ You will receive:
 
 Note the ISO timestamp before any operations.
 
-### Step 2: Start a graded session via MCP (always use MCP for session lifecycle)
+### Step 2: Start a graded session
 
-```
-mutate session start { "grade": true, "name": "grade-<SCENARIO>-<INTERFACE>-run<RUN>", "scope": "global" }
+```bash
+cleo session start --grade --name "grade-<SCENARIO>-run<RUN>" --scope global
 ```
 
 Save the returned `sessionId`.
 
 ### Step 3: Execute scenario operations
 
-Follow the exact operation sequence from the scenario playbook. Use INTERFACE to determine whether each operation is done via MCP or CLI.
-
-**MCP operations** use the query/mutate gateway:
-```
-query tasks find { "status": "active" }
-```
+Follow the exact operation sequence from the scenario playbook. All operations use the CLI.
 
-**CLI operations** use cleo-dev (prefer) or cleo:
 ```bash
 cleo-dev find --status active
 ```
@@ -43,15 +36,14 @@ Scenario sequences are in [../references/scenario-playbook.md](../references/sce
 
 ### Step 4: End the session
 
-```
-mutate session end
+```bash
+cleo session end
 ```
 
 ### Step 5: Grade the session
 
-```
-query check grade { "sessionId": "<saved-id>" }
-# Compatibility alias: query admin grade { "sessionId": "<saved-id>" }
+```bash
+cleo check grade --session "<saved-id>"
 ```
 
 Save the full GradeResult JSON.
@@ -60,7 +52,7 @@ Save the full GradeResult JSON.
 
 Record every operation you executed as a JSONL file. Each line:
 ```json
-{"seq": 1, "gateway": "query", "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "mcp", "timestamp": "..."}
+{"seq": 1, "domain": "tasks", "operation": "find", "params": {}, "success": true, "interface": "cli", "timestamp": "..."}
 ```
 
 ### Step 7: Write output files
@@ -84,10 +76,9 @@ Write to `<OUTPUT_DIR>/<SCENARIO>/arm-<INTERFACE>/`:
 **timing.json** — Fill in what you can; orchestrator fills `total_tokens` and `duration_ms`:
 ```json
 {
-  "arm": "<INTERFACE>",
   "scenario": "<SCENARIO>",
   "run": <RUN_NUMBER>,
-  "interface": "<INTERFACE>",
+  "interface": "cli",
   "executor_start": "<ISO>",
   "executor_end": "<ISO>",
   "executor_duration_seconds": 0,
@@ -124,7 +115,6 @@ Do NOT do these during scenario execution — they will lower the grade intentio
 When complete, summarize:
 ```
 SCENARIO: <id>
-INTERFACE: <interface>
 RUN: <n>
 SESSION_ID: <id>
 TOTAL_SCORE: <n>/100

package/skills/ct-grade/references/ab-test-methodology.md

@@ -14,10 +14,11 @@ An "arm" is a specific test configuration. In CLEO A/B tests, the two most commo
 
 | Arm | Typical Config | Example |
 |-----|---------------|---------|
-| A | MCP gateway | Uses `query`/`mutate` for all operations |
-| B | CLI fallback | Uses `cleo-dev` CLI for equivalent operations |
+| A | Configuration A | Different CLI binary, flags, or prompt setup |
+| B | Configuration B | Alternate setup for comparison |
 
-Arms can also differ by:
+Arms can differ by:
+- CLI binary version (`cleo-dev` vs `cleo`)
 - Session scope (`global` vs `epic:T500`)
 - Tier escalation (with/without `admin.help`)
 - Agent persona (orchestrator vs task-executor)
@@ -71,10 +72,9 @@ save_json(arm_dir + "/timing.json", timing)
 
 ### Why This Matters
 
-Token cost is the primary economic metric for comparing interfaces:
-- MCP operations may use more tokens (richer responses, metadata)
-- CLI operations may use fewer tokens but score lower on S5
-- Score-per-token tells you which interface is more efficient for protocol work
+Token cost is the primary economic metric for comparing configurations:
+- Different configurations may produce different token costs
+- Score-per-token tells you which configuration is more efficient for protocol work
 
 ### Missing Token Data
 
@@ -98,16 +98,16 @@ If you forgot to capture tokens, you cannot recover them. Mark `total_tokens: nu
 | 0-5 pts | Noise level — likely equivalent |
 | 5-15 pts | Meaningful difference — investigate flags |
 | 15-25 pts | Significant — one interface clearly better |
-| 25+ pts | Extreme — likely S5 differential (MCP vs CLI) |
+| 25+ pts | Extreme — likely S5 differential or protocol gap |
 
-### Expected MCP vs CLI Delta
+### Expected Delta
 
 Based on the rubric implementation:
-- S5 Progressive Disclosure: always +20 for MCP (if admin.help called), +10 MCP no help, 0 CLI
+- S5 Progressive Disclosure: +20 if agent uses `admin.help` and follows read-before-write discipline
 - S1-S4: approximately equal if agent follows same protocol steps
-- Total expected delta: **+10 to +20 points** in favor of MCP for equivalent protocols
+- Configuration differences should primarily show up in S5 and token efficiency
 
-If delta exceeds 20 points, investigate whether the CLI agent is also skipping other protocol steps (session.list, descriptions, etc.) due to lack of guidance.
+If delta exceeds 20 points, investigate whether one arm is skipping protocol steps (session.list, descriptions, etc.).
 
 ---
 
@@ -119,8 +119,8 @@ The "git tree" metaphor: each A/B run produces a branch in the results tree. Mul
 ab_results/
   run-001/           ← first full A/B run
     s4/
-      run-01/arm-A/  ← first run, MCP arm
-      run-01/arm-B/  ← first run, CLI arm
+      run-01/arm-A/  ← first run, arm A
+      run-01/arm-B/  ← first run, arm B
       run-01/comparison.json
       run-02/arm-A/
       ...

package/skills/ct-grade/references/domains.md

@@ -1,130 +1,128 @@
 # CLEO Domain Operation Reference for A/B Testing
 
 **Source**: `docs/specs/CLEO-OPERATION-CONSTITUTION.md`
-**Purpose**: Lists the key operations to test in MCP vs CLI A/B comparisons.
+**Purpose**: Lists the key operations to test in A/B comparisons.
+
+All operations use the CLI (`cleo` / `cleo-dev`). There is no MCP interface.
 
 ---
 
-## MCP vs CLI Equivalents
+## CLI Operations by Domain
 
 For each domain, these are the canonical operations to test in A/B mode.
-MCP gateway = audit metadata.gateway is `'query'` or `'mutate'` (set by MCP adapter).
-CLI = operations routed through CLI do NOT set metadata.gateway.
 
 ### tasks (32 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Discovery | `query tasks find { "status": "active" }` | `cleo-dev find --status active` |
-| Show detail | `query tasks show { "taskId": "T123" }` | `cleo-dev show T123` |
-| List children | `query tasks list { "parent": "T100" }` | `cleo-dev list --parent T100` |
-| Create | `mutate tasks add { "title": "...", "description": "..." }` | `cleo-dev add --title "..." --description "..."` |
-| Update | `mutate tasks update { "taskId": "T123", "status": "active" }` | `cleo-dev update T123 --status active` |
-| Complete | `mutate tasks complete { "taskId": "T123" }` | `cleo-dev complete T123` |
-| Exists check | `query tasks exists { "taskId": "T123" }` | `cleo-dev exists T123` |
+| Test Op | CLI |
+|---------|-----|
+| Discovery | `cleo-dev find --status active` |
+| Show detail | `cleo-dev show T123` |
+| List children | `cleo-dev list --parent T100` |
+| Create | `cleo-dev add "title" --description "..."` |
+| Update | `cleo-dev update T123 --status active` |
+| Complete | `cleo-dev complete T123` |
+| Exists check | `cleo-dev exists T123` |
 
-**Key S2 insight**: `tasks.find` (MCP) vs `cleo-dev find` (CLI). Both count toward find:list ratio in the audit log. MCP find at gateway='query', CLI find also logged but without gateway metadata.
+**Key S2 insight**: `cleo-dev find` counts toward find:list ratio in the audit log. Always prefer find over list for discovery.
 
 ### session (19 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Check existing | `query session list` | `cleo-dev session list` |
-| Start | `mutate session start { "grade": true, "scope": "global" }` | `cleo-dev session start --grade --scope global` |
-| End | `mutate session end` | `cleo-dev session end` |
-| Status | `query session status` | `cleo-dev session status` |
-| Record decision | `mutate session record.decision { "decision": "...", "rationale": "..." }` | `cleo-dev session record-decision ...` |
+| Test Op | CLI |
+|---------|-----|
+| Check existing | `cleo-dev session list` |
+| Start | `cleo-dev session start --grade --scope global` |
+| End | `cleo-dev session end` |
+| Status | `cleo-dev session status` |
+| Record decision | `cleo-dev session record-decision --decision "..." --rationale "..."` |
 
-**Critical**: `session.list` (MCP) is what the rubric checks for S1. If CLI does `cleo-dev session list`, it still appears as `domain='session', operation='list'` in the audit log. S1 counts it.
+**Critical**: `session.list` is what the rubric checks for S1. It must appear as `domain='session', operation='list'` in the audit log.
 
-### memory (18 operations) — Tier 1
+### memory (18 operations) -- Tier 1
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Search | `query memory find { "query": "authentication" }` | `cleo-dev memory find "authentication"` |
-| Store observation | `mutate memory observe { "text": "..." }` | `cleo-dev memory observe "..."` |
-| Timeline | `query memory timeline { "anchor": "<id>" }` | N/A (MCP-preferred) |
+| Test Op | CLI |
+|---------|-----|
+| Search | `cleo-dev memory find "authentication"` |
+| Store observation | `cleo-dev observe "..."` |
+| Timeline | `cleo-dev memory timeline <id>` |
 
 ### admin (44 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Dashboard | `query admin dash` | `cleo-dev dash` |
-| Help (S5 key) | `query admin help` | `cleo-dev help` |
-| Grade session | `query admin grade { "sessionId": "<id>" }` | `cleo-dev grade <id>` |
-| Health check | `query admin health` | `cleo-dev health` |
+| Test Op | CLI |
+|---------|-----|
+| Dashboard | `cleo-dev dash` |
+| Help (S5 key) | `cleo-dev help` |
+| Grade session | `cleo-dev check grade --session "<id>"` |
+| Health check | `cleo-dev health` |
 
-**Critical for S5**: Only `query admin help` (MCP) satisfies the `helpCalls` filter in S5. CLI `cleo-dev help` does NOT set `metadata.gateway='query'` or match `domain='admin', operation='help'` — it depends on how the CLI routes internally.
+**Critical for S5**: `cleo-dev help` satisfies the `helpCalls` filter in S5 Progressive Disclosure scoring.
 
-### pipeline (42 operations) — LOOM system
+### pipeline (42 operations) -- LOOM system
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Stage status | `query pipeline stage.status` | `cleo-dev pipeline status` |
-| Stage validate | `query pipeline stage.validate` | `cleo-dev pipeline validate` |
-| Manifest list | `query pipeline manifest.list` | `cleo-dev pipeline manifest list` |
+| Test Op | CLI |
+|---------|-----|
+| Stage status | `cleo-dev pipeline stage.status --epic <id>` |
+| Stage validate | `cleo-dev pipeline stage.validate --epic <id> --stage <stage>` |
+| Manifest list | `cleo-dev manifest list` |
 
 ### check (19 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Test status | `query check test.status` | `cleo-dev check test-status` |
-| Protocol check | `query check protocol` | `cleo-dev check protocol` |
-| Compliance | `query check compliance.summary` | `cleo-dev check compliance` |
+| Test Op | CLI |
+|---------|-----|
+| Test status | `cleo-dev check test-status` |
+| Protocol check | `cleo-dev check protocol` |
+| Compliance | `cleo-dev check compliance` |
 
 ### orchestrate (19 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Status | `query orchestrate status` | `cleo-dev orchestrate status` |
-| Waves | `query orchestrate waves` | `cleo-dev orchestrate waves` |
+| Test Op | CLI |
+|---------|-----|
+| Status | `cleo-dev orchestrator status` |
+| Waves | `cleo-dev orchestrator waves` |
 
 ### tools (32 operations)
 
-| Test Op | MCP | CLI |
-|---------|-----|-----|
-| Skill list (S5 key) | `query tools skill.list` | `cleo-dev tools skill list` |
-| Skill show (S5 key) | `query tools skill.show { "skillId": "ct-cleo" }` | `cleo-dev tools skill show ct-cleo` |
+| Test Op | CLI |
+|---------|-----|
+| Skill list (S5 key) | `cleo-dev skill list` |
+| Skill show (S5 key) | `cleo-dev skill show ct-cleo` |
 
-**S5 note**: `tools.skill.list` and `tools.skill.show` via MCP count toward S5 helpCalls filter.
+**S5 note**: `tools.skill.list` and `tools.skill.show` count toward S5 helpCalls filter.
 
 ---
 
-## A/B Domain Test Configurations
+## A/B Configuration Test Examples
 
 ### Quick A/B: Tasks Domain
 
-**Goal**: Compare MCP vs CLI for core task operations.
-**Operations to execute (both interfaces)**:
-1. `session list` — S1
-2. `tasks find { "status": "active" }` — S2
-3. `tasks show { "taskId": "<valid-id>" }` — S2
-4. `session end` — S1
-
-**Expected score difference**: MCP ~30/100 vs CLI ~20/100 (S5 is 0 for CLI)
+**Goal**: Compare two configurations for core task operations.
+**Operations to execute (both arms)**:
+1. `cleo-dev session list` -- S1
+2. `cleo-dev find --status active` -- S2
+3. `cleo-dev show <valid-id>` -- S2
+4. `cleo-dev session end` -- S1
 
 ### Standard A/B: Full Protocol (S4)
 
-**Goal**: Full lifecycle scenario through both interfaces.
+**Goal**: Full lifecycle scenario through both configurations.
 **Operations**: Follow S4 scenario (10 ops including admin.help).
-**Expected**: MCP 100/100, CLI ~80/100
+**Expected**: 100/100 for protocol-complete arm
 
 ### Targeted A/B: S5 Isolation
 
 **Goal**: Specifically measure the S5 (progressive disclosure) gap.
-**Operations** — same except arm A calls `admin.help`, arm B does not:
+**Operations** -- same except arm A calls `admin.help`, arm B does not:
 
-Arm A (MCP + help):
-```
-query session list → query admin help → query tasks find → mutate session end
+Arm A (with help):
+```bash
+cleo-dev session list && cleo-dev help && cleo-dev find --status active && cleo-dev session end
 ```
 
-Arm B (CLI — no help call):
-```
-cleo-dev session list → cleo-dev find → cleo-dev session end
+Arm B (no help call):
+```bash
+cleo-dev session list && cleo-dev find --status active && cleo-dev session end
 ```
 
-**Expected**: Arm A S5 = 20/20, Arm B S5 = 0/20
+**Expected**: Arm A S5 = 20/20, Arm B S5 = 10/20
 
 ---
 

package/skills/ct-grade/references/grade-spec.md

@@ -152,19 +152,17 @@ helpCalls = entries where:
   OR (domain='tools' AND operation IN ['skill.show','skill.list'])
   OR (domain='skills' AND operation IN ['list','show'])
 
-mcpQueryCalls = entries where metadata.gateway = 'query'
+readOps = entries where operation type is a read (show, find, list, status, etc.)
 ```
 
 | Points | Condition |
 |--------|-----------|
 | +10 | `helpCalls.length > 0` |
-| +10 | `mcpQueryCalls.length > 0` |
+| +10 | `readOps.length > 0` (agent performed read operations before writes) |
 
 **Flags on violation:**
 - `No admin.help or skill lookup calls (load ct-cleo for guidance)`
-- `No MCP query calls (prefer query over CLI for programmatic access)`
-
-**Important**: The `metadata.gateway` field equals `'query'` for MCP query operations. CLI operations do not set this field. This is how MCP vs CLI usage is distinguished in the grade.
+- `No read operations before writes (prefer discovery before mutation)`
 
 ---
 
@@ -218,14 +216,13 @@ interface GradeResult {
 
 ---
 
-## MCP vs CLI Detection in S5
+## S5 Detection
 
-The grading system detects MCP usage via `metadata.gateway === 'query'`. This means:
-- **MCP interface**: All query operations set `metadata.gateway = 'query'` → S5 gets +10
-- **CLI interface**: CLI operations do NOT set metadata.gateway → S5 loses +10
-- **Mixed**: Any single MCP query call is enough for the +10
+The grading system awards S5 points based on:
+1. Presence of `admin.help` or skill lookup calls (+10)
+2. Evidence of read-before-write discipline — agent performed discovery operations before mutations (+10)
 
-This is why A/B tests between MCP and CLI interfaces will reliably show S5 differences.
+All operations use the CLI (`cleo` / `cleo-dev`). There is no MCP interface.
 
 
 ## API Surface Update