@cleocode/skills 2026.5.82 → 2026.5.84

package/skills/ct-grade-v2-1/references/playbook-v2.md

@@ -1,325 +0,0 @@
-# Grade Scenario Playbook v2
-
-Parameterized test scenarios for CLEO grade system validation.
-Updated for CLEO v2026.3+ operation names and 10-domain registry.
-
-All operations use the CLI (`cleo` / `cleo-dev`). There is no MCP interface.
-
----
-
-## Parameterization
-
-All scenarios accept these parameters via run_scenario.py:
-
-| Param | Default | Description |
-|-------|---------|-------------|
-| `--cleo` | `cleo-dev` | CLEO binary to use |
-| `--scope` | `global` | Session scope |
-| `--parent-task` | none | Parent task ID for subtask tests |
-| `--output-dir` | `./grade-results` | Where to save results |
-| `--runs` | 1 | Number of runs (for statistical averaging) |
-| `--seed-task` | none | Pre-existing task ID to work against |
-
----
-
-## S1: Session Discipline
-
-**Rubric target:** S1 Session Discipline 20/20
-
-**Operations (in order):**
-
-```bash
-1. cleo-dev session list
-2. cleo-dev dash
-3. cleo-dev find --status active
-4. cleo-dev show <seed-task>
-5. cleo-dev session end
-```
-
-**Pass criteria:**
-- S1 = 20 (session.list before task ops AND session.end present)
-- S2 >= 15 (only find used, no list)
-- Flags: zero
-
-**Anti-pattern (failing S1 = 0):**
-```bash
-# tasks.find BEFORE session.list
-cleo-dev find --status active
-cleo-dev session list   # too late
-# (no session.end)
-```
-
----
-
-## S2: Task Hygiene
-
-**Rubric target:** S3 Task Hygiene 20/20
-
-**Prerequisites:** `--parent-task` set to an existing task ID.
-
-**Operations:**
-```bash
-1. cleo-dev session list
-2. cleo-dev show <parent-task>                                           # verify parent exists
-3. cleo-dev add "Impl auth" --description "Add JWT auth to API endpoints" --parent <parent-task>
-4. cleo-dev add "Write tests" --description "Unit tests for auth module"
-5. cleo-dev session end
-```
-
-**Pass criteria:**
-- S3 = 20 (all adds have descriptions, parent verified via show)
-- S1 = 20
-- Flags: zero
-
-**Anti-pattern (S3 = 7):**
-```bash
-# No description, no exists check
-cleo-dev add "Impl auth" --parent <id>
-cleo-dev add "Write tests"
-```
-Expected deduction: -5 (no desc task 1) + -5 (no desc task 2) + -3 (no exists check) = 7/20.
-
----
-
-## S3: Error Recovery
-
-**Rubric target:** S4 Error Protocol 20/20
-
-**Prerequisites:** `T99999` does NOT exist.
-
-**Operations:**
-```bash
-1. cleo-dev session list
-2. cleo-dev show T99999                                                  # triggers E_NOT_FOUND (exit code 4)
-3. cleo-dev find "T99999"                                                # recovery lookup (must be within 4 ops)
-4. cleo-dev add "New feature" --description "Feature not found, creating fresh"
-5. cleo-dev session end
-```
-
-**Pass criteria:**
-- S4 = 20 (E_NOT_FOUND followed by recovery; no duplicates)
-- Evidence: `E_NOT_FOUND followed by recovery lookup`
-- Flags: zero
-
-**Anti-pattern (unrecovered, S4 = 15):**
-```bash
-cleo-dev show T99999                               # E_NOT_FOUND
-cleo-dev add "Something" --description "Unrelated"  # NO recovery lookup
-```
-
-**Anti-pattern (duplicates, S4 = 15):**
-```bash
-cleo-dev add "Feature X" --description "First attempt"
-cleo-dev add "Feature X" --description "Second attempt"  # duplicate!
-```
-
----
-
-## S4: Full Lifecycle
-
-**Rubric target:** All 5 dimensions 20/20 (total = 100)
-
-**Prerequisites:** Known task `--seed-task` in pending status.
-
-**Operations (in order):**
-```bash
-1.  cleo-dev session list
-2.  cleo-dev help
-3.  cleo-dev dash
-4.  cleo-dev find --status pending
-5.  cleo-dev show <seed-task>
-6.  cleo-dev update <seed-task> --status active
-    # [agent performs work]
-7.  cleo-dev complete <seed-task>
-8.  cleo-dev find --status pending
-9.  cleo-dev session end --note "Completed <seed-task>"
-```
-
-**Pass criteria:**
-- Total = 100, Grade = A
-- Zero flags
-- Entry count >= 10
-- All 5 dimensions at 20/20
-
----
-
-## S5: Multi-Domain Analysis
-
-**Rubric target:** All 5 dimensions 20/20
-
-**Prerequisites:** `--scope "epic:<parent-task>"` and epic has subtasks.
-
-**Operations:**
-```bash
-1.  cleo-dev session list
-2.  cleo-dev help
-3.  cleo-dev find --parent <parent-task>
-4.  cleo-dev show <subtask-id>
-5.  cleo-dev session context-drift
-6.  cleo-dev session decision-log --task <subtask-id>
-7.  cleo-dev session record-decision --task <subtask-id> --decision "Use adapter pattern" --rationale "Decouples provider logic"
-8.  cleo-dev update <subtask-id> --status active
-9.  cleo-dev complete <subtask-id>
-10. cleo-dev find --parent <parent-task> --status pending
-11. cleo-dev session end
-```
-
-**Pass criteria:**
-- Total = 100, Grade = A
-- Evidence of multi-domain ops (session, tasks, admin)
-- Decision recorded
-
-**Partial variation (S5 = 10 instead of 20):**
-Skip step 2 (`admin.help`). Earns read-before-write +10 but not help/skill +10.
-
----
-
-## S6: Memory Observe & Recall
-
-**Rubric target:** S2 Task Efficiency 15+, S5 Progressive Disclosure 15+
-
-**Operations (in order):**
-```bash
-1. cleo-dev session start --grade --name "grade-s6-memory-observe" --scope global
-2. cleo-dev session list
-3. cleo-dev observe "tasks.find is faster than tasks.list for large datasets" --title "Performance finding"
-4. cleo-dev memory find "tasks.find faster"
-5. cleo-dev memory timeline <returned-id> --before 2 --after 2
-6. cleo-dev memory fetch <id>
-7. cleo-dev session end
-8. cleo-dev check grade --session "<saved-id>"
-```
-
-**Pass criteria:**
-- S5 = 15+ (progressive disclosure via memory ops)
-- S2 = 15+ (find used for retrieval, not broad list)
-- Flags: zero
-
----
-
-## S7: Decision Continuity
-
-**Rubric target:** S1 Session Discipline 20, S5 Progressive Disclosure 15+
-
-**Operations (in order):**
-```bash
-1. cleo-dev session start --grade --name "grade-s7-decision" --scope global
-2. cleo-dev session list
-3. cleo-dev memory decision store "Use adapter pattern for CLI abstraction" --rationale "Decouples interface from business logic" --confidence high
-4. cleo-dev memory decision find "adapter pattern"
-5. cleo-dev memory find "adapter pattern"
-6. cleo-dev memory stats
-7. cleo-dev session end
-8. cleo-dev check grade --session "<saved-id>"
-```
-
-**Pass criteria:**
-- S1 = 20 (session.list before ops)
-- S5 = 15+ (progressive disclosure via memory ops)
-- Flags: zero
-
----
-
-## S8: Pattern & Learning Storage
-
-**Rubric target:** S2 Task Efficiency 15+, S5 Progressive Disclosure 15+
-
-**Operations (in order):**
-```bash
-1. cleo-dev session start --grade --name "grade-s8-patterns" --scope global
-2. cleo-dev session list
-3. cleo-dev memory pattern store "Call session.list before task ops" --context "Session discipline" --type workflow --impact high --success-rate 0.95
-4. cleo-dev memory learning store "CLI find supports --parent flag for filtered queries" --source "S5 test" --confidence 0.9 --actionable
-5. cleo-dev memory pattern find --type workflow --impact high
-6. cleo-dev memory learning find --min-confidence 0.8 --actionable-only
-7. cleo-dev session end
-8. cleo-dev check grade --session "<saved-id>"
-```
-
-**Pass criteria:**
-- S2 = 15+ (pattern.find/learning.find used, not broad list)
-- S5 = 15+ (progressive disclosure via memory ops)
-- Flags: zero
-
----
-
-## S9: NEXUS Cross-Project Ops
-
-**Rubric target:** S5 Progressive Disclosure 20
-
-**Operations (in order):**
-```bash
-1. cleo-dev session start --grade --name "grade-s9-nexus" --scope global
-2. cleo-dev session list
-3. cleo-dev nexus status
-4. cleo-dev nexus list
-5. cleo-dev nexus show <first-project-id>
-6. cleo-dev dash
-7. cleo-dev session end
-8. cleo-dev check grade --session "<saved-id>"
-```
-
-**Pass criteria:**
-- S5 = 20 (cross-domain progressive disclosure)
-- S1 = 20 (session.list first)
-- Note: If nexus list returns empty, skip show and note "no projects registered"
-
----
-
-## S10: Full System Throughput (8 domains)
-
-**Rubric target:** S2 Task Efficiency 15+, S5 Progressive Disclosure 15+
-
-**Operations (in order):**
-```bash
-1.  cleo-dev session start --grade --name "grade-s10-throughput" --scope global
-2.  cleo-dev session list                            # session domain
-3.  cleo-dev help                                    # admin domain
-4.  cleo-dev find --status active                    # tasks domain
-5.  cleo-dev memory find "decisions"                 # memory domain
-6.  cleo-dev nexus status                            # nexus domain
-7.  cleo-dev pipeline stage.status --epic <any-epic-id>  # pipeline domain
-8.  cleo-dev health                                  # check domain
-9.  cleo-dev skill list                              # tools domain
-10. cleo-dev show <from-step-4>
-11. cleo-dev observe "S10 throughput test complete" --title "Throughput"
-12. cleo-dev session end
-13. cleo-dev check grade --session "<saved-id>"
-```
-
-**Pass criteria:**
-- 8 distinct domains hit in audit_log
-- S2 = 15+ (tasks.find used, not tasks.list)
-- S5 = 15+ (progressive disclosure across domains)
-- Flags: zero
-- Note: Step 7 pipeline.stage.status may return E_NOT_FOUND if no epicId — record the attempt, it still logs an audit entry
-
----
-
-## Scoring Quick Reference
-
-| Grade | Threshold | Typical profile |
-|-------|-----------|-----------------|
-| A | >= 90% | All dimensions near max, zero or minimal flags |
-| B | >= 75% | Minor violations in one or two dimensions |
-| C | >= 60% | Several protocol gaps |
-| D | >= 45% | Multiple anti-patterns |
-| F | < 45% | Severe protocol violations |
-
----
-
-## Running Scenarios
-
-```bash
-# Single scenario
-python scripts/run_scenario.py --scenario S3 --cleo cleo-dev
-
-# Full suite
-python scripts/run_scenario.py --scenario full --cleo cleo-dev --output-dir ./results
-
-# With seed task
-python scripts/run_scenario.py --scenario S4 --seed-task T200 --cleo cleo-dev
-
-# Multiple runs for averaging
-python scripts/run_scenario.py --scenario S1 --runs 5 --output-dir ./s1-stats
-```

package/skills/ct-grade-v2-1/references/token-tracking.md

@@ -1,200 +0,0 @@
-# Token Tracking Methodology
-
-How to measure and report token usage for CLEO grade sessions and A/B tests.
-
----
-
-## Measurement Methods (priority order)
-
-### Method 1: Claude Code Agent task notification (canonical)
-
-When running scenarios via Agent tasks, `total_tokens` is available in the task completion notification. This is the actual API token count — the most accurate available.
-
-```python
-# Capture immediately when the agent task completes:
-timing = {
-  "total_tokens": task.total_tokens,    # actual API count
-  "duration_ms": task.duration_ms,
-}
-# Write to timing.json — this data is EPHEMERAL and cannot be recovered later
-```
-
-**Setup:** No configuration needed. Works whenever Claude Code spawns an Agent task.
-
-### Method 2: OTel `claude_code.token.usage`
-
-When Claude Code is configured with OpenTelemetry, actual token counts are available at `~/.cleo/metrics/otel/`.
-
-```bash
-# Enable (add to shell profile):
-export CLAUDE_CODE_ENABLE_TELEMETRY=1
-export OTEL_METRICS_EXPORTER=otlp
-export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
-export OTEL_EXPORTER_OTLP_ENDPOINT="file://${HOME}/.cleo/metrics/otel/"
-```
-
-Fields of interest from `claude_code.token.usage` metric:
-- `input_tokens` — tokens consumed by the prompt
-- `output_tokens` — tokens generated in response
-- `cache_read_input_tokens` — tokens served from cache
-- `session_id` — links to CLEO session
-
-### Method 3: output_chars / 3.5 (JSON estimate)
-
-When neither task notification nor OTel is available, estimate from response character counts:
-
-```
-estimated_tokens ≈ output_chars / 3.5   (JSON responses — denser than prose)
-estimated_tokens ≈ output_chars / 4     (mixed content)
-```
-
-**Accuracy:** ±15-20% for typical JSON responses. Consistent enough for relative comparisons.
-
-### Method 4: Audit entry count (coarse proxy)
-
-Each audit entry represents one operation invocation. As a very rough proxy:
-- One CLI read call ≈ 200–600 tokens total (request + response)
-- One CLI write call ≈ 300–800 tokens total
-
-`entryCount × 150` gives a session-level estimate. Accuracy ±50%.
-
----
-
-## Token Fields in Results
-
-### GRADES.jsonl token metadata
-
-The v2.1 grade scripts append `_tokenMeta` to each grade result:
-
-```json
-{
-  "sessionId": "session-abc123",
-  "totalScore": 85,
-  "_tokenMeta": {
-    "estimationMethod": "otel",
-    "totalEstimatedTokens": 4200,
-    "inputTokens": 3100,
-    "outputTokens": 1100,
-    "cacheReadTokens": 800,
-    "perDomain": {
-      "tasks": 1800,
-      "session": 600,
-      "admin": 400,
-      "memory": 0,
-      "check": 0,
-      "pipeline": 0,
-      "orchestrate": 0,
-      "tools": 200,
-      "nexus": 0,
-      "sticky": 0
-    },
-    "perInterface": {
-      "cli": 3200,
-      "untracked": 1000
-    },
-    "auditEntries": 47,
-    "avgTokensPerEntry": 89
-  }
-}
-```
-
-### A/B test token fields
-
-In `ab-result.json`:
-```json
-{
-  "operation": "tasks.find",
-  "runs": [
-    {
-      "run": 1,
-      "arm_a": {
-        "output_chars": 1240,
-        "estimated_tokens": 310,
-        "duration_ms": 145
-      },
-      "arm_b": {
-        "output_chars": 980,
-        "estimated_tokens": 245,
-        "duration_ms": 88
-      },
-      "token_delta": "+65",
-      "token_delta_pct": "+26.5%"
-    }
-  ]
-}
-```
-
----
-
-## Per-Domain Token Estimation
-
-Typical token ranges per operation type (output only, output_chars / 4):
-
-| Domain | Operation | Typical output_chars | Est. tokens |
-|--------|-----------|---------------------|-------------|
-| tasks | `tasks.find` (10 results) | 2000–4000 | 500–1000 |
-| tasks | `tasks.show` (single) | 800–1500 | 200–375 |
-| tasks | `tasks.list` (full) | 5000–20000+ | 1250–5000+ |
-| session | `session.list` | 1000–3000 | 250–750 |
-| session | `session.status` | 400–800 | 100–200 |
-| admin | `admin.dash` | 1200–2500 | 300–625 |
-| admin | `admin.help` | 2000–5000 | 500–1250 |
-| memory | `memory.find` | 1500–4000 | 375–1000 |
-
-**Key insight:** `tasks.list` is 4–10x more expensive than `tasks.find` for same result set. This is why S2 Discovery Efficiency penalizes list-heavy agents.
-
----
-
-## Using token_tracker.py
-
-```bash
-# Estimate tokens for a specific session from GRADES.jsonl
-python scripts/token_tracker.py \
-  --session-id "session-abc123" \
-  --grades-file .cleo/metrics/GRADES.jsonl
-
-# Use OTEL data if available
-python scripts/token_tracker.py \
-  --session-id "session-abc123" \
-  --otel-dir ~/.cleo/metrics/otel \
-  --grades-file .cleo/metrics/GRADES.jsonl
-
-# Aggregate breakdown across all sessions
-python scripts/token_tracker.py \
-  --grades-file .cleo/metrics/GRADES.jsonl \
-  --breakdown-by domain \
-  --output domain-token-report.json
-
-# Compare two grade sessions
-python scripts/token_tracker.py \
-  --compare session-abc123 session-def456 \
-  --grades-file .cleo/metrics/GRADES.jsonl
-```
-
----
-
-## Token Efficiency Score
-
-The `generate_report.py` script computes a `tokenEfficiencyScore` for each session:
-
-```
-tokenEfficiencyScore = (entriesCompleted / estimatedTokens) * 1000
-```
-
-Higher = more work per token. Use to compare:
-- Pre/post skill improvements
-- Different agent configurations
-- Different CLI binary versions
-
----
-
-## OTEL File Format
-
-OTEL files written by Claude Code are JSONL (one metric per line):
-
-```json
-{"name": "claude_code.token.usage", "value": 4200, "attributes": {"session_id": "...", "model": "claude-sonnet-4-6"}, "timestamp": "2026-03-07T..."}
-{"name": "claude_code.api_request", "value": 1, "attributes": {"input_tokens": 3100, "output_tokens": 1100, "cache_read_input_tokens": 800}, "timestamp": "..."}
-```
-
-`token_tracker.py` reads these files and matches `session_id` to CLEO session IDs stored in `GRADES.jsonl`.