claudecode-omc 5.11.0 → 5.13.0

package/bundled/upstream/ecc/skills/agent-self-evaluation/examples/high-score-example.md

@@ -0,0 +1,87 @@
+# High-Score Example: Adding Retry Logic
+
+## Task
+> Add retry logic with exponential backoff to all HTTP requests in the API client. Max 3 retries, initial delay 1s, max delay 30s.
+
+## Agent Output
+
+### Summary
+Added `httpx.HTTPTransport` with retry configuration to the shared `ApiClient` class. All HTTP methods (GET, POST, PUT, DELETE) now retry on 429, 5xx, and connection errors with exponential backoff. 42 tests pass. Single PR ready to merge.
+
+### Files Changed
+- `src/api_client.py` — Added `RetryTransport` wrapper with `httpx.HTTPTransport(retries=3)` and exponential backoff configured via `httpx.Limits`
+- `tests/test_api_client.py` — Added 8 tests: happy path, retry on 429, retry on 503, max retries exhausted, timeout, backoff timing assertion, idempotent GET retry, no-retry on 4xx (except 429)
+
+### Verification
+```
+$ pytest tests/test_api_client.py -v
+42 passed in 2.34s
+
+$ grep -r "RetryTransport" src/
+src/api_client.py:23: class RetryTransport(httpx.BaseTransport):
+src/api_client.py:45:     self._transport = RetryTransport(...)
+
+$ python -c "from src.api_client import ApiClient; c = ApiClient(); print(type(c._transport))"
+<class 'src.api_client.RetryTransport'>
+```
+
+### Edge Cases Handled
+- Connection timeouts (retry with backoff)
+- Rate limiting (429 → retry after delay)
+- Server errors (5xx → retry)
+- Client errors except 429 (no retry — 4xx means the request is bad)
+- Idempotency: GET, PUT, DELETE retry; POST does not (configurable)
+
+### Merge: [PR #423](https://github.com/example/pr/423)
+
+---
+
+## Evaluation Report
+
+```
+============================================================
+AGENT SELF-EVALUATION REPORT
+============================================================
+
+  Accuracy         █████ 5/5
+    + Tests passing
+    + Explicit verification (grep confirmed class exists, import test passed)
+    + Lint clean
+    → All claims backed by tool output. No hedging.
+
+  Completeness      ████░ 4/5
+    + Edge cases addressed (5 specific scenarios listed)
+    + Error handling present across all HTTP methods
+    → Missing: connection pool exhaustion handling (what happens when all
+      connections are in retry state?) — minor gap, not blocking.
+
+  Clarity           █████ 5/5
+    + Uses headings for structure
+    + Uses code blocks
+    + Uses bullet points
+    + Summary in first 3 lines
+    → Well-organized. Reader can scan in 10 seconds.
+
+  Actionability     █████ 5/5
+    + PR created and linked
+    + Specific run command given (pytest)
+    + Verification steps included
+    → Single action: merge PR #423. Everything else is done.
+
+  Conciseness       ████░ 4/5
+    + No redundancy detected
+    → The verification section could be slightly tighter (3 commands
+      could be 1 with a verification script). Minor.
+
+  OVERALL           4.6/5
+
+TOP IMPROVEMENTS:
+  No axes below 4. Strong output across all dimensions.
+```
+
+### Why This Scores Well
+
+1. **Accuracy pinned to tool output.** Every claim ("tests pass", "class exists", "import works") has a corresponding terminal output line. No "should work" or "probably fine."
+2. **Completeness is explicit about what's covered AND what's not.** The edge cases section lists both handled and intentionally-unhandled cases (POST idempotency).
+3. **Actionability is single-step.** The user only needs to merge one PR. No follow-up tasks, no "then configure X."
+4. **Concision is tight.** The output is ~250 words. The information density is high — every sentence carries weight.

package/bundled/upstream/ecc/skills/agent-self-evaluation/examples/low-score-example.md

@@ -0,0 +1,86 @@
+# Low-Score Example: Adding Retry Logic
+
+## Task
+> Add retry logic with exponential backoff to all HTTP requests in the API client. Max 3 retries, initial delay 1s, max delay 30s.
+
+## Agent Output
+
+Here's a retry implementation. I think this should work for most cases.
+
+I used the urllib3 library since it has built-in retry support. The Retry class handles backoff automatically.
+
+```python
+from urllib3.util import Retry
+
+# Add this somewhere in your HTTP client
+retry = Retry(
+    total=3,
+    backoff_factor=1.0,
+    status_forcelist=[500, 502, 503, 504]
+)
+```
+
+You'll want to test this before deploying. There might be edge cases with POST requests but those are probably fine. I haven't tested the timeout behavior but it should work.
+
+Let me know if you need anything else!
+
+---
+
+## Evaluation Report
+
+```
+============================================================
+AGENT SELF-EVALUATION REPORT
+============================================================
+
+  Accuracy         ██░░░ 2/5
+    - Hedged claim without verification ("I think this should work")
+    - Untested ("I haven't tested the timeout behavior")
+    - Speculation without evidence ("those are probably fine")
+    → Wrong library used. Project uses httpx, not urllib3.
+                      urllib3.util.Retry is incompatible with httpx.
+
+  Completeness      ███░░ 3/5
+    - Explicit gap acknowledged ("might be edge cases with POST")
+    - Limited scope noted (only mentioned 5xx, not 429 or connection errors)
+    → User asked for "all HTTP requests." Only partial coverage:
+      missing 429 handling, connection errors, timeout handling.
+
+  Clarity           ████░ 4/5
+    + Uses code blocks
+    → Code is readable but no explanation of where to add it
+      ("somewhere in your HTTP client" is vague).
+
+  Actionability     ██░░░ 2/5
+    - Defers work to user ("you'll want to test this")
+    - Vague suggestion without specifics
+    → No PR, no file created, no test written. User has to:
+      1. Figure out where to add the code
+      2. Fix the library mismatch (httpx not urllib3)
+      3. Write tests
+      4. Handle POST idempotency
+      5. Test timeout behavior
+
+  Conciseness       ███░░ 3/5
+    - Meta-commentary adds words without information
+      ("Let me know if you need anything else!")
+    → 120 words. Low word count but low information density.
+      Half the text is hedging and disclaimers, not substance.
+
+  OVERALL           2.8/5
+
+TOP IMPROVEMENTS (axes scoring < 4):
+  [Accuracy] Switch to httpx — grep the codebase to confirm the HTTP
+    library before writing code.
+  [Actionability] Create a PR with the changed file + test file. Run the
+    tests. End with "PR #N ready to merge."
+  [Completeness] List what's covered AND what's not. If POST retry is
+    unsafe, say so explicitly with reasoning.
+```
+
+### Why This Scores Poorly
+
+1. **Accuracy fails at the most basic level** — wrong library. One `grep httpx src/` would have caught this. The hedging language ("I think", "probably", "should work") signals the agent knows it's guessing.
+2. **Not actionable.** The user received a code snippet and a list of things they need to do. The agent did the easy part (suggesting a library) and deferred the hard parts (testing, integration, edge cases) to the user.
+3. **Completeness gaps are acknowledged but not fixed.** "Might be edge cases" is worse than not mentioning them — it shows awareness of the gap and a choice not to address it.
+4. **Information density is low.** 120 words, of which ~60 are hedging/disclaimers/politeness. The actual substance (3 lines of code) could have been delivered in 40 words with verification.

package/bundled/upstream/ecc/skills/agent-self-evaluation/references/evaluation-criteria.md

@@ -0,0 +1,71 @@
+# Evaluation Criteria — Detailed Scoring Guide
+
+This reference provides concrete scoring anchors for each axis. Use it when you're unsure whether a gap merits a 4 vs a 3, or a 2 vs a 1.
+
+## Accuracy
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | All facts verified against tool output, docs, or authoritative sources. No errors. | Configured retry via httpx transport — confirmed in httpx docs. All method names verified with grep against codebase. |
+| 4 | One minor inaccuracy that doesn't affect correctness. | Correct library, wrong default value for one parameter (claimed 0.5s, docs say 1.0s). |
+| 3 | One significant factual error, or 3+ minor inaccuracies. | Used `urllib3.Retry` in an httpx codebase. Works in this one case but wrong library. |
+| 2 | Multiple significant errors. Output would fail if followed. | Claimed "add this to package.json" but project uses pyproject.toml. Two other config claims also wrong. |
+| 1 | Fundamentally incorrect. Output contradicts itself or known facts. | Code has syntax errors. API endpoint doesn't exist. Claims a function signature that grep disproves. |
+
+## Completeness
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | All explicit and implicit requirements covered. Edge cases handled. Error paths addressed. | User said "add retry to all HTTP requests." GET, POST, PUT, DELETE all covered. Timeout, 429, 5xx all handled. |
+| 4 | All explicit requirements covered. One implicit requirement missed. | All HTTP methods covered. Forgot to handle connection timeouts (not mentioned but expected). |
+| 3 | One explicit requirement missed, or 2+ implicit gaps. | User said "add logging too." Retry logic added but no logging. |
+| 2 | Multiple explicit requirements missed. Output is a partial solution. | Asked for retry + circuit breaker. Only retry implemented. |
+| 1 | Misses the core request. Delivers something adjacent to what was asked. | Asked for retry logic. Wrote a health check endpoint instead. |
+
+## Clarity
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Perfectly structured. Jargon explained or avoided. Visual hierarchy helps scanning. No ambiguity. | README with clear sections, code blocks, and a 10-second summary at top. |
+| 4 | Generally clear. One section could be better organized or one term undefined. | Good structure but `exponential backoff` used without explanation — assumes the reader knows it. |
+| 3 | Understandable after re-reading. Multiple organizational issues or undefined terms. | The explanation circles the point before getting to it. Several terms used before defined. |
+| 2 | Confusing in places. Reader would need to ask follow-up questions. | Code works but the PR description doesn't explain why retry was needed or what it fixes. |
+| 1 | Unintelligible or contradictory. Reader cannot determine what was done or why. | Output is a wall of text with no structure. Conclusions contradict earlier statements. |
+
+## Actionability
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Single action required. Verification path included. No implicit steps. | "Merge this PR. Tests pass: `42 passed`. Deploy with `./deploy.sh`." |
+| 4 | Single action required but verification path is implied, not explicit. | "Merge this PR." (Tests exist but weren't cited. User has to check themselves.) |
+| 3 | Multiple actions required, or one action with unclear next step. | "Review and merge. Then update the config." (Which config? Where? No link or path.) |
+| 2 | User must figure out how to use the output. Missing critical instructions. | Code written but no test file, no run instructions, no PR created. User has to assemble everything. |
+| 1 | Output cannot be acted on without significant rework or clarification. | "Here's a design idea." (No code, no file, no PR. User has to start from scratch.) |
+
+## Conciseness
+
+| Score | Anchor | Example |
+|---|---|---|
+| 5 | Every sentence earns its place. No redundancy. Information density is high. | 30 lines that say what 60 lines would. No repeated points. No filler. |
+| 4 | Minor redundancy. One paragraph could be tightened. | Good overall but repeats the motivation in both the PR description and code comments. |
+| 3 | Noticeable redundancy. 20%+ of content could be removed without loss. | Explains the same concept three times (in summary, body, and conclusion). Verbose examples. |
+| 2 | Significantly bloated. 40%+ of content is filler or repetition. | 200 lines for a task that needed 60. Restates the user's question. Includes irrelevant background. |
+| 1 | Noise-to-signal ratio is inverted. More filler than substance. | 500-line response to a 2-line question. Most of it is boilerplate, repetition, or irrelevant context. |
+
+## Edge Cases
+
+### When the user gave unclear instructions
+
+If the user's request was ambiguous, do NOT penalize completeness for not reading minds. Instead, note in the evaluation: "User's request was ambiguous about [scope]. I chose interpretation [chosen interpretation]. If they meant [alternative interpretation], this score would drop to [score]."
+
+### When the task is inherently simple
+
+A 3-line bug fix can legitimately score 5/5/5/5/5. The rubric scales with complexity — a simple task done perfectly IS a 5.0. Don't invent gaps to justify lower scores.
+
+### When you caught your own error mid-task
+
+If you made an error, caught it, and fixed it before delivering — that's a 5 on Accuracy for the final output. The evaluation is about what the user received, not your internal process. Note the self-correction as evidence of thoroughness, not as a penalty.
+
+### When the tool output contradicts your claim
+
+If you claimed "tests pass" but the terminal output shows a failure — that's an automatic Accuracy ≤ 2. Tool output is ground truth. Claims without verification are the most common source of low accuracy scores.

package/bundled/upstream/ecc/skills/agent-self-evaluation/references/hook-integration.md

@@ -0,0 +1,64 @@
+# Hook Integration for Session-Stop Self-Evaluation
+
+Add this hook to `hooks/hooks.json` to remind the agent to self-evaluate at the end of every session (the hook echoes a reminder; it does not run the evaluator automatically):
+
+```json
+{
+  "hooks": {
+    "Stop": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "echo '[Self-Eval] Session complete. Consider running agent-self-evaluation to rate your output.'"
+          }
+        ],
+        "description": "Remind agent to self-evaluate at session end"
+      }
+    ]
+  }
+}
+```
+
+`Stop` events do not require a `matcher` field (it is optional for `Stop`, `Notification`, `UserPromptSubmit`, and `SubagentStop` per `scripts/ci/validate-hooks.js`). If omitted, the hook object only needs `hooks` and metadata such as `description`.
+
+## Integration with the Python Evaluator
+
+The `scripts/evaluate.py` script can be used as a standalone tool:
+
+```bash
+# Pipe agent output directly
+echo "Your agent response here" | python3 skills/agent-self-evaluation/scripts/evaluate.py
+
+# From files
+python3 skills/agent-self-evaluation/scripts/evaluate.py --task task.txt --output response.txt
+```
+
+To integrate it into hooks, capture the last agent output to a file first, then run the evaluator. For lightweight reminders after shell-based verification, use a simple supported matcher string:
+
+```json
+{
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "Bash",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "echo '[Self-Eval] If this command completed verification for a non-trivial task, consider running agent-self-evaluation.'"
+          }
+        ],
+        "description": "Remind agent to self-evaluate after shell verification"
+      }
+    ]
+  }
+}
+```
+
+This avoids documenting unsupported command-expression matcher syntax. If your harness supports command-level matcher expressions, prefer a word-boundary regex such as `\b(pytest|npm test|go test)\b` rather than a broad `test` substring.
+
+These hooks are opt-in. Add them to your local `hooks/hooks.json` if you want automated evaluation prompts.
+
+## Manual Usage (Recommended)
+
+The most reliable approach is manual invocation — the agent runs self-evaluation as part of its workflow when the `agent-self-evaluation` skill is active, without requiring hook configuration. The skill's "When to Activate" section already covers trigger conditions (multi-file changes, debugging sessions, design documents).