@uzysjung/agent-harness 26.151.0 → 26.152.0

package/templates/skills/eval-harness/SKILL.md

@@ -1,306 +0,0 @@
----
-name: eval-harness
-description: Formal evaluation framework for Claude Code sessions implementing eval-driven development (EDD) principles
-origin: ECC
-tools: Read, Write, Edit, Bash, Grep, Glob
----
-
-# Eval Harness Skill
-
-A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles.
-
-## When to Activate
-
-- Setting up eval-driven development (EDD) for AI-assisted workflows
-- Defining pass/fail criteria for Claude Code task completion
-- Measuring agent reliability with pass@k metrics
-- Creating regression test suites for prompt or agent changes
-- Benchmarking agent performance across model versions
-
-## Philosophy
-
-Eval-Driven Development treats evals as the "unit tests of AI development":
-- Define expected behavior BEFORE implementation
-- Run evals continuously during development
-- Track regressions with each change
-- Use pass@k metrics for reliability measurement
-
-## Eval Types
-
-### Capability Evals
-Test if Claude can do something it couldn't before:
-```markdown
-[CAPABILITY EVAL: feature-name]
-Task: Description of what Claude should accomplish
-Success Criteria:
-  - [ ] Criterion 1
-  - [ ] Criterion 2
-  - [ ] Criterion 3
-Expected Output: Description of expected result
-```
-
-### Regression Evals
-Ensure changes don't break existing functionality:
-```markdown
-[REGRESSION EVAL: feature-name]
-Baseline: SHA or checkpoint name
-Tests:
-  - existing-test-1: PASS/FAIL
-  - existing-test-2: PASS/FAIL
-  - existing-test-3: PASS/FAIL
-Result: X/Y passed (previously Y/Y)
-```
-
-## Grader Types
-
-### 1. Code-Based Grader
-Deterministic checks using code:
-```bash
-# Check if file contains expected pattern
-grep -q "export function handleAuth" src/auth.ts && echo "PASS" || echo "FAIL"
-
-# Check if tests pass
-npm test -- --testPathPattern="auth" && echo "PASS" || echo "FAIL"
-
-# Check if build succeeds
-npm run build && echo "PASS" || echo "FAIL"
-```
-
-### 2. Model-Based Grader
-Use Claude to evaluate open-ended outputs:
-```markdown
-[MODEL GRADER PROMPT]
-Evaluate the following code change:
-1. Does it solve the stated problem?
-2. Is it well-structured?
-3. Are edge cases handled?
-4. Is error handling appropriate?
-
-Score: 1-5 (1=poor, 5=excellent)
-Reasoning: [explanation]
-```
-
-### 3. Human Grader
-Flag for manual review:
-```markdown
-[HUMAN REVIEW REQUIRED]
-Change: Description of what changed
-Reason: Why human review is needed
-Risk Level: LOW/MEDIUM/HIGH
-```
-
-## Metrics
-
-### pass@k
-"At least one success in k attempts"
-- pass@1: First attempt success rate
-- pass@3: Success within 3 attempts
-- Typical target: pass@3 > 90%
-
-### pass^k
-"All k trials succeed"
-- Higher bar for reliability
-- pass^3: 3 consecutive successes
-- Use for critical paths
-
-## Eval Workflow
-
-### 1. Define (Before Coding)
-
-Write the spec to a file (`.claude/evals/<feature>.md`) before implementing. Give every eval a
-stable ID — `C1..Cn` for capability, `R1..Rn` for regression — so the same identifier carries
-from definition to the post-implementation status line, and a reviewer can check them off one by
-one. Prose-only lists ("Can create new user account") can't be referenced or scored.
-
-````markdown
-# EVAL: <feature name> (<phase/PR>)
-
-**Feature**: <one line>
-**Baseline**: commit <sha>          # what "regression" is measured against
-**Target**: pass@1 = 100% for capability evals
-
-## Capability Evals
-
-### C1: <name>
-- <concrete, checkable expectation — inputs → expected output>
-
-### C2: <name>
-- <expectation>
-
-## Regression Evals
-
-### R1: <existing behavior that must not move>
-- <expectation>
-
-## Test Command
-```bash
-pytest tests/test_<area>.py -v -k "<selector>"
-```
-
-## Status (after implementation)
-- C1-Cn: PASS via <tests (N개) / route check / manual>
-- R1-Rn: PASS
-
-**Overall: pass@1 = <x>%**
-````
-
-Two fields carry most of the weight. **Baseline commit** makes "regression" falsifiable — without
-it, R-evals are opinions about the past. **Test Command** makes the spec re-runnable by someone
-who didn't write it; an eval nobody can re-run is documentation, not a gate.
-
-Fill the Status section *after* implementing, in the same file. A spec whose status is still empty
-at merge time means the evals were written and never used.
-
-### 2. Implement
-Write code to pass the defined evals.
-
-### 3. Evaluate
-```bash
-# Run capability evals
-[Run each capability eval, record PASS/FAIL]
-
-# Run regression evals
-npm test -- --testPathPattern="existing"
-
-# Generate report
-```
-
-### 4. Report
-```markdown
-EVAL REPORT: feature-xyz
-========================
-
-Capability Evals:
-  create-user:     PASS (pass@1)
-  validate-email:  PASS (pass@2)
-  hash-password:   PASS (pass@1)
-  Overall:         3/3 passed
-
-Regression Evals:
-  login-flow:      PASS
-  session-mgmt:    PASS
-  logout-flow:     PASS
-  Overall:         3/3 passed
-
-Metrics:
-  pass@1: 67% (2/3)
-  pass@3: 100% (3/3)
-
-Status: READY FOR REVIEW
-```
-
-## Integration Patterns
-
-### Pre-Implementation
-```
-/eval define feature-name
-```
-Creates eval definition file at `.claude/evals/feature-name.md`
-
-### During Implementation
-```
-/eval check feature-name
-```
-Runs current evals and reports status
-
-### Post-Implementation
-```
-/eval report feature-name
-```
-Generates full eval report
-
-## Eval Storage (.md + .log Pair Format)
-
-각 평가 항목은 **`<topic>.md` (설계) + `<topic>.log` (실행 결과)** 쌍으로 저장. 강제. 단독 .md만 있으면 재현 불가.
-
-```
-.claude/
-  evals/
-    feature-xyz.md        # Eval definition (Capability/Regression/Test 3섹션 필수)
-    feature-xyz.log       # Eval run history (실행 시각, grader, pass/fail)
-    session-YYYYMMDD.md   # 세션 단위 회고 + 차기 backlog
-    session-YYYYMMDD.log  # 동일 세션의 grader 출력
-    baseline.json         # Regression baselines (선택)
-```
-
-> eval 산출물은 `docs/evals/*.{md,log}` 로 모은다 — 실행 로그와 판정을 같은 자리에 둔다.
-
-### .md 파일 의무 섹션 (3개)
-
-```markdown
-# Eval: <topic>
-
-## Capability
-[새 능력 — Claude/agent가 무엇을 할 수 있는지]
-- AC: [측정 가능 기준]
-- Grader: code-based / model-based / human
-
-## Regression
-[기존 기능 보호 — 변경으로 깨지면 안 되는 baseline]
-- Baseline: <SHA or checkpoint>
-- Tests: [목록]
-
-## Test
-[실행 절차 — 누가 다시 돌려도 동일 결과 나와야 함]
-- Setup: [사전 조건]
-- Run: `bash run-eval.sh <topic>` 또는 명시적 명령
-- Expected: [기대 출력]
-```
-
-### .log 파일 형식
-
-각 실행마다 append. 시간순 누적.
-
-```
-=== 2026-04-19 14:32 (run #1) ===
-Capability: 3/3 PASS (pass@1)
-Regression: 5/5 PASS (pass^3)
-Status: SHIP READY
-
-=== 2026-04-20 09:15 (run #2 — after refactor) ===
-Capability: 3/3 PASS
-Regression: 4/5 PASS (login-flow regressed at SHA abc123)
-Status: BLOCKED — fix login-flow first
-```
-
-## Best Practices
-
-1. **Define evals BEFORE coding** - Forces clear thinking about success criteria
-2. **Run evals frequently** - Catch regressions early
-3. **Track pass@k over time** - Monitor reliability trends
-4. **Use code graders when possible** - Deterministic > probabilistic
-5. **Human review for security** - Never fully automate security checks
-6. **Keep evals fast** - Slow evals don't get run
-7. **Version evals with code** - Evals are first-class artifacts
-
-## Example: Adding Authentication
-
-```markdown
-## EVAL: add-authentication
-
-### Phase 1: Define (10 min)
-Capability Evals:
-- [ ] User can register with email/password
-- [ ] User can login with valid credentials
-- [ ] Invalid credentials rejected with proper error
-- [ ] Sessions persist across page reloads
-- [ ] Logout clears session
-
-Regression Evals:
-- [ ] Public routes still accessible
-- [ ] API responses unchanged
-- [ ] Database schema compatible
-
-### Phase 2: Implement (varies)
-[Write code]
-
-### Phase 3: Evaluate
-Run: /eval check add-authentication
-
-### Phase 4: Report
-EVAL REPORT: add-authentication
-==============================
-Capability: 5/5 passed (pass@3: 100%)
-Regression: 3/3 passed (pass^3: 100%)
-Status: SHIP IT
-```

package/templates/skills/eval-harness/agents/openai.yaml

@@ -1,7 +0,0 @@
-interface:
-  display_name: "Eval Harness"
-  short_description: "Eval-driven development with pass/fail criteria"
-  brand_color: "#EC4899"
-  default_prompt: "Set up eval-driven development with pass/fail criteria"
-policy:
-  allow_implicit_invocation: true

package/templates/skills/verification-loop/SKILL.md

@@ -1,250 +0,0 @@
----
-name: verification-loop
-description: >-
-  A comprehensive verification system. Selects and runs proportional verification tracks for UI,
-  API/service, CLI/TUI, library/SDK, documents/configuration, and real user flows, then ends every
-  run with a fixed verdict — PASS / PASS_WITH_NITS / FAIL — plus severity-labeled findings
-  (CRITICAL/HIGH/MEDIUM/LOW) and the evidence each one rests on. Use after implementation, before
-  a PR or handoff, after a refactor, or to verify a claimed fix. Do NOT use a green build, a passing
-  type check, or file existence as proof of user-visible completion, and do NOT let the instance
-  that wrote the change issue its own verdict.
-origin: ECC
----
-
-# Verification Loop Skill
-
-> Derived from the `verification-loop` skill in everything-claude-code (ECC), used under the MIT
-> License.
-
-A comprehensive verification system for coding sessions. The job is not "run the gates" — it is to
-**verify the changed behavior through the surface a real user or consumer actually uses**, and to
-end with one verdict that cannot be softened into prose.
-
-## When to Use
-
-Invoke this skill:
-- After completing a feature or significant code change
-- Before creating a PR
-- When you want to ensure quality gates pass
-- After refactoring
-- To verify that a fix actually closed the reported failure
-
-### Positive triggers
-
-- "Verify this implementation before handoff."
-- "Confirm the bug is closed in the real CLI."
-- "Run visual and functional QA on the changed flow."
-
-### Negative triggers
-
-- Pure planning with no artifact to verify.
-- A generic request for more tests with no changed behavior or acceptance criterion in hand.
-- Anything where you would be verifying code you just wrote yourself (see the Verdict Contract).
-
-## Pick the real surface first
-
-Static gates support verification; they do not constitute it. Before running anything, list the
-acceptance criteria as observable outcomes and select every track the change touches — UI,
-API/service, CLI/TUI, library/SDK, documents/configuration, user flow. Each track has a required
-observation and its own evidence record: read
-[references/tracks.md](references/tracks.md).
-
-## Then pick the depth — it scales with the risk
-
-The Testing rule says depth follows risk and names what counts as high-risk (authentication,
-authorization, payments and settlement, personal data, data integrity, concurrency, state
-transitions, migrations). It deliberately stops there. **Which** instruments to widen with is a
-per-change judgment, and this is where that menu lives:
-
-| Instrument | Reach for it when |
-|---|---|
-| Regression beyond the directly affected scope | the change moves a shared type, a schema, a config default, or anything the affected scope was only *assumed* to bound |
-| Integration / contract tests | it crosses a boundary someone else owns — a service, a queue, a stored format, a published API |
-| Critical-path E2E | a user-visible flow that must not break can only be observed end to end |
-| Mutation testing | **you doubt the tests you already have would catch a defect** |
-
-**Mutation testing is an option, not a requirement.** It is expensive, so being labelled
-high-risk is not by itself a reason to run it — the reason is uncertainty about detection power.
-When the existing suite has already been shown to bite (a negative control, a caught regression),
-the doubt it answers is not there and the cost buys nothing.
-
-Two things this depth choice is *not*: it is not a coverage target, and it is not the scheduled
-full run. Full regression, full E2E, full mutation, and periodic security scanning belong to the
-CI/CD schedule — do not launch them here for one change.
-
-## Verification Phases (static gates)
-
-### Phase 1: Build Verification
-```bash
-# Check if project builds
-npm run build 2>&1 | tail -20
-# OR
-pnpm build 2>&1 | tail -20
-```
-
-If build fails, STOP and fix — then re-verify from Phase 1 in a fresh instance. Continuing
-through the remaining phases yourself would make you the verifier of code you just wrote,
-which the Verdict Contract below forbids.
-
-### Phase 2: Type Check
-```bash
-# TypeScript projects
-npx tsc --noEmit 2>&1 | head -30
-
-# Python projects
-pyright . 2>&1 | head -30
-```
-
-Report all type errors. Fix critical ones before continuing.
-
-### Phase 3: Lint Check
-```bash
-# JavaScript/TypeScript
-npm run lint 2>&1 | head -30
-
-# Python
-ruff check . 2>&1 | head -30
-```
-
-### Phase 4: Test Suite
-```bash
-# Run tests with coverage
-npm run test -- --coverage 2>&1 | tail -50
-
-# Check coverage threshold
-# Target: 80% minimum
-```
-
-Report:
-- Total tests: X
-- Passed: X
-- Failed: X
-- Coverage: X%
-
-Where the project declares its own threshold, that number wins — 80% is the floor to use when no
-project threshold exists, not a licence to lower one that does.
-
-### Phase 5: Security Scan
-```bash
-# Check for secrets
-grep -rn "sk-" --include="*.ts" --include="*.js" . 2>/dev/null | head -10
-grep -rn "api_key" --include="*.ts" --include="*.js" . 2>/dev/null | head -10
-
-# Check for console.log
-grep -rn "console.log" --include="*.ts" --include="*.tsx" src/ 2>/dev/null | head -10
-```
-
-### Phase 6: Diff Review
-```bash
-# Show what changed
-git diff --stat
-git diff HEAD~1 --name-only
-```
-
-Review each changed file for:
-- Unintended changes
-- Missing error handling
-- Potential edge cases
-
-Adapt the commands to the repository's stack — the six phases (build, types, lint, tests,
-security, diff) are the contract; `npm`/`pyright`/`ruff` are just this list's defaults.
-
-## Then run the live surface
-
-Static green with no live run verifies nothing a user can see. For each selected track:
-
-- **UI** — browser-driven flow, screenshots, console errors, responsive states, and explicit
-  approval before any intentional baseline change.
-- **API/service** — start the service and call the real endpoint; check response *and* side effect.
-- **CLI/TUI** — invoke the built command through its terminal interface; check exit code, stdout,
-  and the error path.
-- **Library/SDK** — run a minimal consumer program against the public interface.
-- **Documents/configuration** — parse, resolve references, and exercise the consumer that loads it.
-- **User flow** — complete the representative end-to-end scenario across the affected components.
-
-Do not reuse evidence from a different path: one path's green is not another path's evidence.
-
-## Output Format
-
-After running all phases, produce a verification report:
-
-```
-VERIFICATION REPORT
-==================
-
-Build:     [PASS/FAIL]
-Types:     [PASS/FAIL] (X errors)
-Lint:      [PASS/FAIL] (X warnings)
-Tests:     [PASS/FAIL] (X/Y passed, Z% coverage)
-Security:  [PASS/FAIL] (X issues)
-Diff:      [X files changed]
-
-Live surface: [track] — [command/interaction] → [observed] (artifact: path)
-
-Verdict:   PASS | PASS_WITH_NITS | FAIL
-
-Findings:
-| ID | Severity | Finding | Evidence (file:line / command output) |
-|----|----------|---------|---------------------------------------|
-| F1 | HIGH     | ...     | ...                                   |
-```
-
-Skipped and unverified criteria are listed explicitly, with the reason — silence reads as "passed".
-
-## Verdict Contract
-
-The report ends with exactly one verdict. Free-prose closings ("looks ready", "should be
-fine") are banned — they leave room to bury defects. A fixed vocabulary makes the report
-honest and machine-checkable.
-
-| Verdict | Meaning | Action |
-|---------|---------|--------|
-| **PASS** | All gates green, zero findings at any severity | Ship |
-| **PASS_WITH_NITS** | Ship-safe: only LOW/MEDIUM findings, each recorded with a follow-up | Ship + log follow-ups |
-| **FAIL** | Any gate red, or one or more CRITICAL/HIGH findings | Block → fix → **re-verify** |
-
-Every finding gets exactly one severity:
-
-- **CRITICAL** — data loss, security hole, or the change misbehaves in real use if shipped
-- **HIGH** — main-path defect or regression; users will hit it
-- **MEDIUM** — edge-case or quality defect; unlikely to block real use
-- **LOW** — nit: style, naming, doc wording
-
-Rules:
-- Severity is judged by impact evidence, not by how easy the fix is.
-- FAIL → fix → re-verify is one cycle. A fix alone never upgrades the verdict — the
-  re-verification must reproduce green.
-- A run that aborts early (Phase 1 build failure) still emits a report: verdict **FAIL**
-  with the failing gate as a CRITICAL finding. Stopping to fix is how you *reach* the next
-  verdict, not a reason to skip issuing this one — an unreported run reads as "not run".
-- Missing evidence for a required criterion is a FAIL, not a PASS with a caveat.
-- The instance that wrote the change never issues its own verdict: verification runs in a
-  fresh instance (see the model-orchestration skill's V&V separation).
-
-## Safety
-
-- Never kill broad process patterns, hardcode credentials, or bypass authentication controls.
-- Never update a visual baseline without explicit review of the changed images.
-- Do not claim installed, supported, or tested runtimes that were not actually exercised.
-- Tests, browsers, and services create local artifacts and processes — keep them inside the
-  requested workspace and stop the ones you started.
-- Stop before authentication, baseline replacement, or any external mutation the user did not
-  authorize.
-
-## Continuous Mode
-
-For long sessions, run verification every 15 minutes or after major changes:
-
-```markdown
-Set a mental checkpoint:
-- After completing each function
-- After finishing a component
-- Before moving to next task
-
-Run: /verify
-```
-
-## Integration with Hooks
-
-This skill complements PostToolUse hooks but provides deeper verification.
-Hooks catch issues immediately; this skill provides comprehensive review.

package/templates/skills/verification-loop/agents/openai.yaml

@@ -1,7 +0,0 @@
-interface:
-  display_name: "Verification Loop"
-  short_description: "Build, test, lint, typecheck verification"
-  brand_color: "#10B981"
-  default_prompt: "Run verification: build, test, lint, typecheck, security"
-policy:
-  allow_implicit_invocation: true

package/templates/skills/verification-loop/references/tracks.md

@@ -1,49 +0,0 @@
-# Verification tracks
-
-## Select the real surface
-
-| Track | Required observation |
-|---|---|
-| UI | Browser flow, rendered state, console, responsive layout, accessibility-relevant interaction |
-| API or service | Running process, real request, response and side effect |
-| CLI or TUI | Built command, exit code, stdout or screen behavior, error path |
-| Library or SDK | Minimal consumer program using the public interface |
-| Document or configuration | Parser or loader result, resolved references, consumer behavior |
-| User flow | End-to-end outcome across the affected components |
-
-Static checks (SKILL.md Phases 1-6) support these tracks but do not replace them. A change that
-touches two tracks needs evidence from both — one track's green is not the other's evidence.
-
-## UI baseline policy
-
-- Capture deterministic viewports and named states.
-- Compare with the approved baseline using hashes or pixel comparison before semantic review.
-- Treat blank pages, missing core content, and console errors as regressions.
-- Require human approval before adopting an intentional changed baseline.
-- Never launch browsers by killing broad process patterns, and never embed login credentials.
-
-## Evidence record
-
-For each acceptance criterion record:
-
-- exact command or interaction;
-- environment and relevant version;
-- observed outcome;
-- exit status;
-- artifact or screenshot path;
-- `pass` / `fail` / `skipped` / `unverified`;
-- why this evidence covers this criterion.
-
-The last field is the one that catches self-deception: evidence that cannot be tied to a criterion
-is output, not proof. `skipped` and `unverified` belong in the report — an omitted criterion reads
-as a passing one.
-
-## Verdict
-
-- `PASS` — all required gates and user-surface scenarios passed with no findings.
-- `PASS_WITH_NITS` — required behavior passed; only recorded LOW/MEDIUM follow-ups remain.
-- `FAIL` — any required gate failed, a main path is broken, or evidence is missing for a required
-  criterion.
-
-Severity definitions and the rules that bind a verdict to them are in SKILL.md "Verdict Contract";
-they are not restated here so the two cannot drift apart.