aw-ecc 1.4.32 → 1.4.48

package/skills/aw-adk/evals/eval-delete-skill.md

@@ -0,0 +1,90 @@
+---
+name: eval-delete-skill
+target: skill/aw-adk
+category: functional
+difficulty: intermediate
+---
+
+# Eval: Delete Skill — Reverse Reference Cleanup in Agents
+
+## Task
+
+Test that deleting a skill also finds and cleans up agents that reference it in their `skills:` frontmatter, preventing phantom dependencies.
+
+### Prompt
+
+```
+First, create a temporary skill called temp-delete-test-patterns in the platform/data namespace. It teaches temporary testing patterns for data pipelines. It needs no scripts or references — just a simple SKILL.md.
+
+Then create a temporary agent called temp-data-tester in the platform/data namespace. Tools: Read, Grep. Model: haiku. Skills: [platform-data-temp-delete-test-patterns]. Description: "Temporary agent that uses the temp skill."
+
+After both are created, delete the skill temp-delete-test-patterns using the ADK delete flow. When warned about the agent reference, confirm you want to clean it up too. Confirm deletion when prompted.
+```
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Namespace** | `platform/data` |
+| **Domain** | `data` |
+| **Target artifact** | `skills/aw-adk/SKILL.md` |
+| **Target type** | `skill` (create then delete) |
+
+## Expected Outcomes
+
+- [ ] **Skill created** at `.aw/.aw_registry/platform/data/skills/temp-delete-test-patterns/SKILL.md`
+- [ ] **Agent created** referencing the skill in `skills:` frontmatter
+- [ ] **Delete flow initiated** for the skill
+- [ ] **Reverse reference scan** — finds the agent that references this skill
+- [ ] **Warning shown** — "temp-data-tester references this skill in its skills: frontmatter"
+- [ ] **User asked** whether to clean up the reference
+- [ ] **Skill file + evals deleted**
+- [ ] **Agent's skills: frontmatter updated** — reference to the deleted skill removed
+- [ ] **No phantom dependencies remain** — agent no longer references a non-existent skill
+- [ ] **`aw link` ran**
+
+## Grading Criteria
+
+### PASS
+
+- All 10 outcomes met
+- Agent file still exists but no longer references the deleted skill
+
+### PARTIAL
+
+- Skill deleted but agent's skills: frontmatter not updated (phantom created)
+- OR no reverse reference scan performed
+
+### FAIL
+
+- Skill not deleted
+- Agent also deleted (overkill — should only remove the reference)
+- No warning about the dependent agent
+
+## Evaluation Method
+
+**Type:** hybrid
+
+### Deterministic Checks
+
+```bash
+# Skill should be gone
+test ! -d ".aw/.aw_registry/platform/data/skills/temp-delete-test-patterns" || echo "FAIL: skill still exists"
+
+# Agent should still exist
+test -f ".aw/.aw_registry/platform/data/agents/temp-data-tester.md" || echo "FAIL: agent was deleted (should only clean reference)"
+
+# Agent should NOT reference the deleted skill
+grep -q "temp-delete-test-patterns" ".aw/.aw_registry/platform/data/agents/temp-data-tester.md" 2>/dev/null && echo "FAIL: phantom reference in agent"
+```
+
+### Model-Based Checks
+
+- Did the ADK warn about the agent dependency before deleting?
+- Did it offer to clean up the reference rather than silently deleting?
+
+## Baseline Expectations
+
+- Without ADK: Skill deleted, agent left with phantom reference that breaks at runtime.
+- With ADK: Reverse reference scan catches the dependency, cleans it up.
+- **Expected delta:** 0 phantom references with ADK vs. 1+ without

package/skills/aw-adk/evals/eval-meta-eval-coverage.md

@@ -0,0 +1,78 @@
+---
+name: eval-meta-eval-coverage
+target: skill/aw-adk
+category: structural
+difficulty: intermediate
+---
+
+# Eval: Meta-Eval — Scenario Coverage
+
+## Task
+
+Test that evals created by the ADK cover both happy path AND failure scenarios — not just happy-path-only. The ADK's eval gate requires "happy path + at least one failure scenario." This meta-eval verifies that requirement is actually met.
+
+### Prompt
+
+```
+Create a skill for Redis caching patterns in the platform/data namespace.
+```
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Namespace** | `platform/data` |
+| **Domain** | `data` |
+| **Target artifact** | evals created by ADK during skill creation |
+| **Target type** | `eval` (meta) |
+
+## Expected Outcomes
+
+- [ ] **Skill created** with 2+ colocated evals
+- [ ] **At least one happy-path eval** — tests the skill working correctly with valid input
+- [ ] **At least one failure-scenario eval** — tests error handling, edge cases, or invalid input
+- [ ] **Failure eval is not just "minimal input"** — it tests a genuinely different scenario (not the happy path with fewer words)
+- [ ] **Eval purposes are distinct** — the two evals test meaningfully different aspects, not the same scenario with different wording
+- [ ] **Each eval has PASS/FAIL criteria** that are independently verifiable
+
+## Grading Criteria
+
+### PASS
+
+- 2+ evals exist
+- At least one is clearly a failure/edge-case scenario (not a relabeled happy path)
+- Each has distinct, verifiable pass/fail criteria
+
+### PARTIAL
+
+- 2+ evals exist but both are variations of happy path
+- OR only 1 eval created
+
+### FAIL
+
+- No evals created
+- OR all evals test the same scenario
+
+## Evaluation Method
+
+**Type:** hybrid
+
+### Deterministic Checks
+
+```bash
+# Verify 2+ eval files
+EVAL_COUNT=$(ls .aw/.aw_registry/platform/data/skills/redis-caching-patterns/evals/eval-*.md 2>/dev/null | wc -l)
+[[ "$EVAL_COUNT" -ge 2 ]] || echo "FAIL: fewer than 2 evals"
+```
+
+### Model-Based Checks
+
+- Read each eval's scenario: are they testing genuinely different cases?
+- Does at least one eval describe a failure condition (invalid input, missing data, error state)?
+- Would a broken skill pass all evals? (If yes → insufficient coverage)
+
+## Baseline Expectations
+
+- Without ADK: Single happy-path eval or no evals at all.
+- With ADK: 2+ evals with distinct scenarios covering happy path and failure.
+- **Expected delta:** 100% coverage of both paths with ADK

package/skills/aw-adk/evals/eval-meta-eval-determinism.md

@@ -0,0 +1,81 @@
+---
+name: eval-meta-eval-determinism
+target: skill/aw-adk
+category: behavioral
+difficulty: advanced
+---
+
+# Eval: Meta-Eval — Scoring Determinism
+
+## Task
+
+Test that the ADK's scoring produces consistent results. The same artifact scored twice should receive the same tier and similar per-dimension scores. Flaky scoring undermines trust in the entire rubric system.
+
+### Prompt (run twice)
+
+```
+Score this skill: .aw/.aw_registry/platform/data/skills/redis-caching-patterns/SKILL.md
+```
+
+Run the exact same scoring prompt twice against the same artifact. Compare the two score outputs.
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Target artifact** | any existing skill with stable content |
+| **Target type** | `skill` |
+
+## Expected Outcomes
+
+- [ ] **Both runs produce a 10-dimension score table**
+- [ ] **Same tier in both runs** — if run 1 is B-Tier, run 2 must also be B-Tier
+- [ ] **Per-dimension scores within ±1 point** — a dimension scored 7 in run 1 should be 6-8 in run 2
+- [ ] **Total score within ±5 points** — e.g., 72 and 76 is acceptable; 72 and 85 is not
+- [ ] **Same improvement suggestions** — the top 3 gaps identified should overlap between runs
+- [ ] **Both runs reference rubric-skill.md** — scoring is rubric-based, not ad-hoc
+
+## Grading Criteria
+
+### PASS
+
+- Same tier in both runs
+- Total score difference ≤ 5 points
+- Per-dimension scores within ±1
+- Top 3 improvement suggestions overlap (at least 2 of 3 match)
+
+### PARTIAL
+
+- Same tier but total score difference 6-10 points
+- OR different tier but adjacent (B vs C, not B vs D)
+
+### FAIL
+
+- Different tiers separated by 2+ levels (e.g., B-Tier vs D-Tier)
+- Total score difference > 10 points
+- Improvement suggestions are completely different between runs
+
+## Evaluation Method
+
+**Type:** hybrid
+
+### Deterministic Checks
+
+```bash
+# Compare total scores from both runs (requires parsing the output)
+# Tier must match exactly
+# Total must be within ±5
+```
+
+### Model-Based Checks
+
+- Extract the total score from each run's output
+- Compare tier assignments
+- Compare per-dimension scores
+- Compare improvement suggestions for overlap
+
+## Baseline Expectations
+
+- Without ADK: Scoring is ad-hoc, varies wildly between runs (±20+ points).
+- With ADK: Rubric-anchored scoring with ≤5 point variance.
+- **Expected delta:** 75%+ reduction in score variance

package/skills/aw-adk/evals/eval-meta-eval-false-pass.md

@@ -0,0 +1,81 @@
+---
+name: eval-meta-eval-false-pass
+target: skill/aw-adk
+category: behavioral
+difficulty: advanced
+---
+
+# Eval: Meta-Eval — False Pass Resistance
+
+## Task
+
+Test that evals created by the ADK can actually detect bad artifacts. The ADK creates an agent, then creates evals for it. Then a known-bad version of the agent (missing critical sections, wrong structure) is fed to those evals. The evals must FAIL the bad agent — not give it a false pass.
+
+This is a meta-eval: it tests the quality of evals that the ADK produces, not the ADK's create flow itself.
+
+### Prompt (two-step)
+
+**Step 1:** Create an agent for log analysis in the platform/infra namespace.
+
+**Step 2:** Take the evals that were just created. Run them against this known-bad agent:
+
+```markdown
+---
+name: log-analyzer
+description: "Analyzes logs"
+---
+
+# Log Analyzer
+Looks at logs and finds problems.
+```
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Namespace** | `platform/infra` |
+| **Target artifact** | evals created by ADK in step 1 |
+| **Target type** | `eval` (meta) |
+
+## Expected Outcomes
+
+- [ ] **Step 1 completes** — a well-structured agent is created with evals
+- [ ] **Known-bad agent is structurally deficient** — missing: tools, model, category, squad, skills, identity section, core mission, critical rules, process, deliverables
+- [ ] **Evals FAIL the known-bad agent** — at least 1 eval produces a FAIL verdict
+- [ ] **Failure reasons are specific** — "missing Identity section" not just "low quality"
+- [ ] **Evals don't false-pass** — a clearly deficient agent must not get PASS or even PARTIAL
+
+## Grading Criteria
+
+### PASS
+
+- At least 1 eval FAILs the known-bad agent
+- Failure reasons reference specific missing sections or frontmatter fields
+- The well-structured agent from step 1 would PASS the same evals
+
+### PARTIAL
+
+- Evals give PARTIAL (not PASS) to the known-bad agent
+- Some discrimination but not full rejection
+
+### FAIL
+
+- Evals PASS the known-bad agent (false pass)
+- OR evals can't be run against the bad agent (no mechanism)
+- OR evals only check surface features (file exists, has frontmatter) that the bad agent satisfies
+
+## Evaluation Method
+
+**Type:** model-based
+
+### Model-Based Checks
+
+- Do the evals contain assertions that the known-bad agent would fail?
+- Are assertions specific enough to distinguish good from bad?
+- Would substituting the bad agent into the eval's expected outcomes produce FAIL?
+
+## Baseline Expectations
+
+- Without ADK: Evals are always-pass stubs that accept any output.
+- With ADK: Evals have discriminating assertions that catch missing sections and thin content.
+- **Expected delta:** 80%+ false-pass detection rate with ADK evals

package/skills/aw-adk/evals/eval-score-accuracy.md

@@ -0,0 +1,95 @@
+---
+name: eval-score-accuracy
+target: skill/aw-adk
+category: behavioral
+difficulty: intermediate
+---
+
+# Eval: Score Accuracy — Deliberately Minimal Agent Gets Low Score
+
+## Task
+
+Test that the ADK's scoring is calibrated — a deliberately minimal agent (missing most sections) should receive a D-Tier score, not be generously graded. This targets the self-scoring bias where the LLM creates an artifact and then scores it high.
+
+### Prompt
+
+```
+Score this agent: .aw/.aw_registry/revex/reselling/backend/agents/stub-agent.md
+```
+
+### Setup (pre-condition)
+
+Create a minimal stub agent before running the eval:
+
+```markdown
+---
+name: stub-agent
+description: "Does stuff"
+tools: [Read]
+---
+
+# Stub Agent
+
+This agent helps with things.
+```
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Namespace** | `revex/reselling` |
+| **Domain** | `backend` |
+| **Target artifact** | `skills/aw-adk/SKILL.md` |
+| **Target type** | `agent` |
+
+## Expected Outcomes
+
+- [ ] **ADK reads rubric-agent.md** before scoring
+- [ ] **Score table has 10 dimensions** — not a summary number
+- [ ] **Total score is D-Tier (below 40)** — the stub agent is missing: identity section, core mission, critical rules, process, deliverables, communication style, success metrics, model field, category field, squad field
+- [ ] **Missing dimensions identified** — the score output names specific gaps (not "looks good overall")
+- [ ] **No inflation** — scores for missing sections are 0, not 3-5 "because the section could be implied"
+- [ ] **Improvement suggestions provided** — concrete next steps to raise the score
+
+## Grading Criteria
+
+### PASS
+
+- Total score is D-Tier (0-39)
+- At least 6 dimensions scored 0 (missing entirely)
+- Missing sections explicitly named
+
+### PARTIAL
+
+- Total score is C-Tier (40-59) — some inflation but identifies gaps
+- OR correct D-Tier but fewer than 6 zero-scored dimensions
+
+### FAIL
+
+- Total score is B-Tier or above (60+) — severe inflation
+- OR no per-dimension breakdown (just a summary score)
+- OR does not read the rubric before scoring
+
+## Evaluation Method
+
+**Type:** hybrid
+
+### Deterministic Checks
+
+```bash
+# Verify the score output contains a table with 10 rows
+# (model-based check needed to parse the actual scores)
+```
+
+### Model-Based Checks
+
+- Is the total score below 40?
+- Are missing sections scored 0 (not given partial credit)?
+- Did the executor read rubric-agent.md before scoring?
+- Are improvement suggestions specific (not "add more content")?
+
+## Baseline Expectations
+
+- Without ADK: Model says "looks good, 7/10" with no rubric reference.
+- With ADK: Calibrated score using rubric-agent.md, D-Tier for stub, specific gaps identified.
+- **Expected delta:** 30+ point difference in score accuracy

package/skills/aw-adk/evals/eval-type-redirect.md

@@ -0,0 +1,68 @@
+---
+name: eval-type-redirect
+target: skill/aw-adk
+category: behavioral
+difficulty: advanced
+---
+
+# Eval: Type Redirect — Command Request That Should Be a Skill
+
+## Task
+
+Test that the ADK's type classifier catches misclassifications. The prompt asks to "create a command" but the subject matter (static knowledge, best practices) is actually a skill. The ADK should redirect or at minimum flag the mismatch during the interview.
+
+### Prompt
+
+```
+Create a command for React best practices in the platform/frontend namespace. It should cover component patterns, hooks usage, state management, and performance optimization tips.
+```
+
+## Context
+
+| Field | Value |
+|-------|-------|
+| **Namespace** | `platform/frontend` |
+| **Domain** | `frontend` |
+| **Target artifact** | `skills/aw-adk/SKILL.md` |
+| **Target type** | `skill` (despite user saying "command") |
+
+## Expected Outcomes
+
+- [ ] **Type redirect detected** — the ADK recognizes "React best practices" is static knowledge (skill), not a multi-phase workflow (command)
+- [ ] **User informed of redirect** — explains why this is a skill, not a command (commands automate workflows with agents and phases; skills encode knowledge)
+- [ ] **Proceeds as skill** — after redirect, follows the skill create flow
+- [ ] **OR asks user to confirm** — "This sounds like a skill (knowledge reference) rather than a command (workflow). Should I create it as a skill?"
+- [ ] **Does NOT blindly create a command** — a "React best practices command" with forced phases and agent roster would be the wrong artifact type
+
+## Grading Criteria
+
+### PASS
+
+- Redirect detected and communicated to user
+- Proceeds with correct type (skill) after confirmation
+
+### PARTIAL
+
+- Creates the artifact but notes during interview that it might be a skill
+- OR creates a skill without explaining the redirect
+
+### FAIL
+
+- Creates a command with forced multi-phase structure for static knowledge
+- No mention of type mismatch
+
+## Evaluation Method
+
+**Type:** model-based
+
+### Model-Based Checks
+
+- Did the executor question the "command" classification?
+- Did it explain the difference between commands (workflow) and skills (knowledge)?
+- Did it ultimately create a skill (or ask user to choose)?
+
+## Baseline Expectations
+
+- Without ADK: Creates whatever the user asked for literally — a forced "command" with fake phases.
+- With ADK: Type classifier catches the mismatch and redirects.
+- **Expected delta:** correct type 90%+ with ADK vs. literal compliance without

package/skills/aw-adk/evals/evals.json

@@ -0,0 +1,96 @@
+{
+  "artifact_name": "aw-adk",
+  "artifact_type": "skill",
+  "evals": [
+    {
+      "name": "eval-create-skill",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "create-mode"
+    },
+    {
+      "name": "eval-create-agent",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "create-mode"
+    },
+    {
+      "name": "eval-create-command",
+      "category": "functional",
+      "difficulty": "advanced",
+      "group": "create-mode"
+    },
+    {
+      "name": "eval-create-rule",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "create-mode"
+    },
+    {
+      "name": "eval-create-eval",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "create-mode"
+    },
+    {
+      "name": "eval-type-redirect",
+      "category": "behavioral",
+      "difficulty": "advanced",
+      "group": "cross-cutting"
+    },
+    {
+      "name": "eval-score-accuracy",
+      "category": "behavioral",
+      "difficulty": "intermediate",
+      "group": "cross-cutting"
+    },
+    {
+      "name": "eval-colocated-placement",
+      "category": "structural",
+      "difficulty": "basic",
+      "group": "cross-cutting"
+    },
+    {
+      "name": "eval-meta-eval-false-pass",
+      "category": "behavioral",
+      "difficulty": "advanced",
+      "group": "meta-evals"
+    },
+    {
+      "name": "eval-meta-eval-coverage",
+      "category": "structural",
+      "difficulty": "intermediate",
+      "group": "meta-evals"
+    },
+    {
+      "name": "eval-meta-eval-determinism",
+      "category": "behavioral",
+      "difficulty": "advanced",
+      "group": "meta-evals"
+    },
+    {
+      "name": "eval-delete-agent",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "delete-mode"
+    },
+    {
+      "name": "eval-delete-rule",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "delete-mode"
+    },
+    {
+      "name": "eval-delete-skill",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "delete-mode"
+    },
+    {
+      "name": "eval-delete-command",
+      "category": "functional",
+      "difficulty": "intermediate",
+      "group": "delete-mode"
+    }
+  ]
+}