@event4u/agent-config 8.9.0 → 8.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +189 -0
- package/README.md +5 -5
- package/dist/agent-src/commands/challenge-me/vision.md +121 -32
- package/dist/agent-src/commands/challenge-me/with-docs.md +21 -8
- package/dist/agent-src/commands/commit/in-chunks.md +1 -1
- package/dist/agent-src/commands/commit.md +2 -1
- package/dist/agent-src/commands/cost/report.md +14 -3
- package/dist/agent-src/commands/feature/dev.md +4 -4
- package/dist/agent-src/commands/humanize.md +101 -0
- package/dist/agent-src/commands/jira-ticket.md +2 -2
- package/dist/agent-src/commands/memory/load.md +15 -0
- package/dist/agent-src/commands/optimize/project.md +12 -0
- package/dist/agent-src/commands/roadmap/create.md +12 -11
- package/dist/agent-src/commands/tests/create.md +29 -7
- package/dist/agent-src/commands/work.md +33 -0
- package/dist/agent-src/commands/worktree/cleanup.md +83 -0
- package/dist/agent-src/commands/worktree/create.md +64 -0
- package/dist/agent-src/commands/worktree/status.md +75 -0
- package/dist/agent-src/commands/worktree/verify.md +70 -0
- package/dist/agent-src/commands/worktree.md +85 -0
- package/dist/agent-src/contexts/execution/autonomy-mechanics.md +66 -0
- package/dist/agent-src/contexts/execution/orchestration-benchmark-gate.md +1 -0
- package/dist/agent-src/contexts/execution/orchestration-telemetry.md +91 -34
- package/dist/agent-src/contexts/execution/roadmap-process-loop.md +14 -0
- package/dist/agent-src/contexts/execution/roadmap-writing-source-derived.md +65 -0
- package/dist/agent-src/contexts/execution/subagent-modes-detail.md +84 -0
- package/dist/agent-src/rules/autonomous-execution.md +7 -37
- package/dist/agent-src/rules/broken-access-control.md +7 -45
- package/dist/agent-src/rules/code-comment-discipline.md +148 -0
- package/dist/agent-src/rules/context-hygiene.md +14 -0
- package/dist/agent-src/rules/delegation-policy.md +10 -8
- package/dist/agent-src/rules/design-fidelity.md +44 -0
- package/dist/agent-src/rules/git-history-discipline.md +4 -79
- package/dist/agent-src/rules/legal-safety-floor.md +11 -55
- package/dist/agent-src/rules/roadmap-progress-sync.md +5 -56
- package/dist/agent-src/rules/spreadsheet-source-quality.md +61 -0
- package/dist/agent-src/rules/think-before-action.md +10 -0
- package/dist/agent-src/rules/verify-before-complete.md +12 -0
- package/dist/agent-src/skills/accessibility-auditor/evals/evals.json +15 -0
- package/dist/agent-src/skills/agent-docs-writing/SKILL.md +13 -12
- package/dist/agent-src/skills/ai-council/SKILL.md +19 -9
- package/dist/agent-src/skills/analysis-skill-router/evals/evals.json +13 -0
- package/dist/agent-src/skills/api-design/SKILL.md +7 -5
- package/dist/agent-src/skills/api-testing/SKILL.md +16 -2
- package/dist/agent-src/skills/authz-review/SKILL.md +59 -2
- package/dist/agent-src/skills/blast-radius-analyzer/SKILL.md +1 -1
- package/dist/agent-src/skills/bug-analyzer/SKILL.md +19 -16
- package/dist/agent-src/skills/character-consistency/evals/evals.json +16 -0
- package/dist/agent-src/skills/code-refactoring/SKILL.md +9 -9
- package/dist/agent-src/skills/command-routing/evals/evals.json +14 -0
- package/dist/agent-src/skills/content-funnel-design/SKILL.md +2 -0
- package/dist/agent-src/skills/dashboard-design/evals/evals.json +16 -0
- package/dist/agent-src/skills/data-flow-mapper/SKILL.md +1 -1
- package/dist/agent-src/skills/dcf-modeling/evals/evals.json +20 -0
- package/dist/agent-src/skills/decision-record/evals/evals.json +20 -0
- package/dist/agent-src/skills/defense-in-depth/SKILL.md +2 -2
- package/dist/agent-src/skills/dependency-upgrade/SKILL.md +7 -7
- package/dist/agent-src/skills/design-intelligence/SKILL.md +8 -8
- package/dist/agent-src/skills/design-intelligence/evals/evals.json +14 -0
- package/dist/agent-src/skills/design-review/SKILL.md +29 -2
- package/dist/agent-src/skills/design-system-capture/SKILL.md +21 -0
- package/dist/agent-src/skills/design-system-capture/evals/evals.json +15 -0
- package/dist/agent-src/skills/developer-like-execution/evals/evals.json +16 -0
- package/dist/agent-src/skills/doc-coauthoring/SKILL.md +29 -0
- package/dist/agent-src/skills/doc-coauthoring/evals/evals.json +16 -0
- package/dist/agent-src/skills/editorial-calendar/SKILL.md +2 -0
- package/dist/agent-src/skills/editorial-calendar/evals/domain-truth.json +16 -0
- package/dist/agent-src/skills/editorial-calendar/evals/evals.json +16 -0
- package/dist/agent-src/skills/estimate-ticket/evals/evals.json +16 -0
- package/dist/agent-src/skills/existing-ui-audit/SKILL.md +30 -0
- package/dist/agent-src/skills/existing-ui-audit/evals/triggers.json +47 -0
- package/dist/agent-src/skills/fe-design/SKILL.md +43 -14
- package/dist/agent-src/skills/fe-design/evals/triggers.json +47 -0
- package/dist/agent-src/skills/forecasting/evals/evals.json +20 -0
- package/dist/agent-src/skills/fundraising-narrative/evals/domain-truth.json +16 -0
- package/dist/agent-src/skills/fundraising-narrative/evals/evals.json +16 -0
- package/dist/agent-src/skills/git-workflow/SKILL.md +86 -0
- package/dist/agent-src/skills/html-deck/SKILL.md +27 -0
- package/dist/agent-src/skills/humanizer/SKILL.md +147 -0
- package/dist/agent-src/skills/humanizer/data/patterns.md +190 -0
- package/dist/agent-src/skills/humanizer/evals/triggers.json +17 -0
- package/dist/agent-src/skills/iconography/SKILL.md +24 -0
- package/dist/agent-src/skills/incident-commander/evals/domain-truth.json +16 -0
- package/dist/agent-src/skills/incident-commander/evals/evals.json +20 -0
- package/dist/agent-src/skills/laravel/SKILL.md +14 -1
- package/dist/agent-src/skills/launch-readiness/SKILL.md +20 -19
- package/dist/agent-src/skills/launch-readiness/evals/evals.json +21 -0
- package/dist/agent-src/skills/legal-practice-profile/SKILL.md +126 -0
- package/dist/agent-src/skills/logging-monitoring/SKILL.md +1 -1
- package/dist/agent-src/skills/logging-monitoring/evals/evals.json +16 -0
- package/dist/agent-src/skills/mcp/SKILL.md +1 -1
- package/dist/agent-src/skills/memory-consolidation/SKILL.md +14 -1
- package/dist/agent-src/skills/messaging-architecture/evals/domain-truth.json +16 -0
- package/dist/agent-src/skills/messaging-architecture/evals/evals.json +16 -0
- package/dist/agent-src/skills/migration-architect/SKILL.md +2 -1
- package/dist/agent-src/skills/perf-feedback-craft/evals/evals.json +16 -0
- package/dist/agent-src/skills/performance/SKILL.md +1 -1
- package/dist/agent-src/skills/pest-testing/SKILL.md +1 -1
- package/dist/agent-src/skills/php-debugging/SKILL.md +17 -0
- package/dist/agent-src/skills/playwright-testing/SKILL.md +16 -0
- package/dist/agent-src/skills/prompt-engineering-patterns/evals/evals.json +16 -0
- package/dist/agent-src/skills/readme-writing/SKILL.md +2 -0
- package/dist/agent-src/skills/readme-writing-package/SKILL.md +2 -0
- package/dist/agent-src/skills/refine-prompt/evals/evals.json +16 -0
- package/dist/agent-src/skills/refine-ticket/evals/evals.json +16 -0
- package/dist/agent-src/skills/release-comms/SKILL.md +2 -0
- package/dist/agent-src/skills/release-comms/evals/evals.json +16 -0
- package/dist/agent-src/skills/rice-prioritization/evals/evals.json +20 -0
- package/dist/agent-src/skills/roadmap-writing/SKILL.md +22 -51
- package/dist/agent-src/skills/runway-cognition/evals/evals.json +20 -0
- package/dist/agent-src/skills/scenario-modeling/evals/evals.json +20 -0
- package/dist/agent-src/skills/security-audit/SKILL.md +6 -9
- package/dist/agent-src/skills/sentry-integration/SKILL.md +2 -2
- package/dist/agent-src/skills/skill-improvement-pipeline/SKILL.md +8 -0
- package/dist/agent-src/skills/source-discovery/SKILL.md +26 -24
- package/dist/agent-src/skills/spreadsheet-authoring/SKILL.md +98 -0
- package/dist/agent-src/skills/spreadsheet-authoring/evals/triggers.json +47 -0
- package/dist/agent-src/skills/subagent-orchestration/SKILL.md +55 -46
- package/dist/agent-src/skills/subagent-orchestration/evals/evals.json +42 -0
- package/dist/agent-src/skills/systematic-debugging/SKILL.md +10 -0
- package/dist/agent-src/skills/systematic-debugging/evals/evals.json +20 -0
- package/dist/agent-src/skills/test-case-discovery/SKILL.md +118 -0
- package/dist/agent-src/skills/test-case-discovery/evals/triggers.json +52 -0
- package/dist/agent-src/skills/test-driven-development/SKILL.md +32 -6
- package/dist/agent-src/skills/test-driven-development/evals/evals.json +16 -0
- package/dist/agent-src/skills/testing-anti-patterns/SKILL.md +7 -6
- package/dist/agent-src/skills/threat-modeling/evals/evals.json +21 -0
- package/dist/agent-src/skills/typography-system/evals/evals.json +14 -0
- package/dist/agent-src/skills/unit-economics-modeling/evals/evals.json +24 -0
- package/dist/agent-src/skills/using-git-worktrees/SKILL.md +10 -0
- package/dist/agent-src/skills/verify-completion-evidence/evals/evals.json +16 -0
- package/dist/agent-src/skills/vision-articulation/evals/evals.json +16 -0
- package/dist/agent-src/skills/voice-and-tone-design/evals/evals.json +16 -0
- package/dist/agent-src/skills/worktree-lifecycle/SKILL.md +187 -0
- package/dist/agent-src/skills/worktree-lifecycle/evals/evals.json +38 -0
- package/dist/agent-src/skills/worktree-lifecycle/evals/triggers.json +52 -0
- package/dist/agent-src/templates/agent-settings.md +3 -2
- package/dist/agent-src/templates/agents/agent-project-settings.example.yml +15 -1
- package/dist/agent-src/templates/roadmaps.md +14 -0
- package/dist/cli/commands/recordTriggerEval.js +4 -14
- package/dist/cli/commands/recordTriggerEval.js.map +1 -1
- package/dist/discovery/deprecation-report.md +1 -1
- package/dist/discovery/discovery-manifest.json +374 -88
- package/dist/discovery/discovery-manifest.json.sha256 +1 -1
- package/dist/discovery/discovery-manifest.summary.md +9 -9
- package/dist/discovery/orphan-report.md +1 -1
- package/dist/discovery/packs.json +30 -18
- package/dist/discovery/trust-report.md +5 -5
- package/dist/discovery/workspaces.json +31 -15
- package/dist/install/install.mjs +12 -7
- package/dist/mcp/registry-manifest.json +4 -4
- package/dist/router.json +1 -1
- package/dist/scripts/_lib/claude_settings_hooks.js +8 -0
- package/dist/scripts/_lib/claude_settings_hooks.js.map +1 -1
- package/dist/scripts/_lib/trigger_eval_floors.js +24 -0
- package/dist/scripts/_lib/trigger_eval_floors.js.map +1 -0
- package/dist/scripts/condense.js +12 -2
- package/dist/scripts/condense.js.map +1 -1
- package/dist/server/schemas/settings.js +5 -2
- package/dist/server/schemas/settings.js.map +1 -1
- package/docs/CLAIMS.md +24 -3
- package/docs/architecture.md +3 -3
- package/docs/benchmark.md +51 -0
- package/docs/catalog.md +23 -9
- package/docs/command-flows.md +7 -1
- package/docs/contracts/CHANGELOG-conventions.md +6 -0
- package/docs/contracts/ai-council-config.md +92 -0
- package/docs/contracts/command-clusters.md +1 -0
- package/docs/contracts/design-artifact-lifecycle.md +93 -0
- package/docs/contracts/design-artifact-verification.md +30 -1
- package/docs/contracts/evidence-based-pruning.md +1 -0
- package/docs/contracts/hook-architecture-v1.md +25 -0
- package/docs/contracts/kernel-membership.md +35 -0
- package/docs/contracts/measurement-baseline.md +1 -0
- package/docs/contracts/release-pr-gating.md +26 -0
- package/docs/contracts/release-sizing.md +107 -0
- package/docs/contracts/surface-agent-contracts.md +153 -0
- package/docs/contracts/write-engine.md +38 -1
- package/docs/customization.md +12 -0
- package/docs/decisions/ADR-002-kernel-bucket-overrides.md +11 -0
- package/docs/decisions/ADR-003-flat-cluster-subs-and-colon-syntax.md +11 -0
- package/docs/decisions/ADR-006-skill-tools-python-pilot.md +1 -1
- package/docs/decisions/ADR-028-root-layout.md +1 -1
- package/docs/decisions/ADR-038-canonical-settings-path.md +10 -0
- package/docs/decisions/ADR-092-defer-command-tier-alias-removal.md +29 -5
- package/docs/decisions/ADR-093-ai-council-config-user-global.md +1 -1
- package/docs/decisions/ADR-100-global-knowledge-card-sharing.md +7 -0
- package/docs/decisions/ADR-103-global-knowledge-default-off-until-measured.md +2 -2
- package/docs/decisions/ADR-105-automatic-subagent-orchestration.md +1 -1
- package/docs/decisions/ADR-114-commit-policy-iron-law-override.md +11 -0
- package/docs/decisions/ADR-115-command-cluster-consolidation-phase-4.md +1 -1
- package/docs/decisions/ADR-116-memory-tripwire-activation-path.md +11 -1
- package/docs/decisions/ADR-117-subagents-auto-default-on.md +2 -1
- package/docs/decisions/ADR-118-loop-engineering-boundaries.md +162 -0
- package/docs/decisions/ADR-119-global-knowledge-default-on.md +104 -0
- package/docs/decisions/ADR-200-python-to-typescript-migration.md +11 -1
- package/docs/decisions/ADR-rule-kernel-and-router.md +11 -0
- package/docs/decisions/INDEX.md +11 -9
- package/docs/design/cross-model-parity-eval.md +82 -0
- package/docs/design/length-neutral-judge-rerun.md +92 -0
- package/docs/distribution/consumer-matrix.md +96 -0
- package/docs/featured-skills.md +2 -2
- package/docs/getting-started-by-role.md +1 -1
- package/docs/getting-started.md +2 -2
- package/docs/governance-advantage.md +1 -1
- package/docs/guidelines/agent-infra/roadmap-progress-mechanics.md +80 -5
- package/docs/guidelines/code-clarity.md +106 -1
- package/docs/guidelines/design-antipatterns.md +20 -0
- package/docs/guidelines/php/php-coding-patterns.md +12 -2
- package/docs/proof.md +21 -15
- package/docs/release-runbook.md +3 -0
- package/docs/self-review-gate.md +51 -0
- package/docs/wedge/production-validator/production-validator.md +27 -0
- package/package.json +1 -1
- package/src/config/agent-settings.template.yml +29 -9
- package/src/scripts/_cli/cmd_doctor.ts +34 -1
- package/src/scripts/_lib/auto_dispatch.corpus.test.ts +87 -0
- package/src/scripts/_lib/claude_settings_hooks.ts +8 -0
- package/src/scripts/_lib/knowledge_global.ts +6 -2
- package/src/scripts/_lib/knowledge_global_redaction.ts +42 -4
- package/src/scripts/_lib/lexical_index.ts +15 -2
- package/src/scripts/_lib/orchestration_record.ts +27 -0
- package/src/scripts/_lib/orchestration_savings.ts +52 -0
- package/src/scripts/_lib/trigger_eval_floors.ts +31 -0
- package/src/scripts/ai_council/chairman.ts +90 -0
- package/src/scripts/ai_council/clients.ts +18 -3
- package/src/scripts/ai_council/config.ts +157 -0
- package/src/scripts/ai_council/debate_gates.ts +89 -0
- package/src/scripts/ai_council/orchestrator.ts +65 -2
- package/src/scripts/ai_council/prompts.ts +121 -11
- package/src/scripts/ai_council/replay.ts +17 -0
- package/src/scripts/ai_council/stance_tally.ts +184 -0
- package/src/scripts/ai_tells_rules.ts +433 -0
- package/src/scripts/bench_humanizer_eval.ts +240 -0
- package/src/scripts/build_discovery_manifest.ts +6 -0
- package/src/scripts/build_proof.ts +21 -9
- package/src/scripts/check_claims.ts +40 -0
- package/src/scripts/check_release_adjacent_health.ts +121 -0
- package/src/scripts/check_rule_invariants.ts +282 -0
- package/src/scripts/condense.ts +16 -2
- package/src/scripts/consumer_matrix.ts +392 -0
- package/src/scripts/council_cli.ts +147 -0
- package/src/scripts/detect_ai_tells.ts +312 -0
- package/src/scripts/generate_index.ts +3 -2
- package/src/scripts/install.ts +12 -0
- package/src/scripts/knowledge_global_cli.ts +1 -1
- package/src/scripts/lint_changelog_rollback.ts +293 -0
- package/src/scripts/lint_framework_leakage.ts +88 -7
- package/src/scripts/lint_framework_leakage_allowlist.json +59 -572
- package/src/scripts/lint_knowledge_scale.ts +30 -0
- package/src/scripts/lint_originality_shingles.ts +210 -0
- package/src/scripts/orchestration_record.ts +17 -0
- package/src/scripts/orchestration_savings_report.ts +36 -3
- package/src/scripts/schemas/agent-settings.schema.json +11 -0
- package/src/scripts/second_brain_retrieval.ts +44 -2
- package/src/scripts/self_review_gate.test.ts +65 -0
- package/src/scripts/self_review_gate.ts +298 -0
- package/src/scripts/skill_eval_coverage.ts +71 -3
- package/src/scripts/skill_linter.ts +1 -0
- package/src/scripts/trigger_eval_grandfather.json +223 -224
- package/src/scripts/trigger_eval_rotation.ts +344 -0
- package/src/scripts/update_counts.ts +5 -17
- package/src/scripts/worktree_cleanup_check.ts +263 -0
- package/src/server/schemas/settings.ts +8 -3
- package/src/scripts/analysis_freshness.ts +0 -362
- package/src/scripts/export_replay_corpus.ts +0 -208
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "spreadsheet-authoring",
|
|
3
|
+
"last_eval": "2026-07-10",
|
|
4
|
+
"description": "5 should-trigger + 5 should-not-trigger queries for the spreadsheet surface floor (road-to-surface-specific-agent-contracts Phase 2). Should-trigger covers building/editing a spreadsheet or model, entering formulas or sourced/financial figures, and charting raw data. Should-not-trigger covers near-miss neighbours whose vocabulary overlaps but belong to a different surface/skill: unit-economics reasoning, runway shape, a prose document, a slide deck, and backend work.",
|
|
5
|
+
"queries": [
|
|
6
|
+
{
|
|
7
|
+
"q": "build a 3-statement financial model in a spreadsheet",
|
|
8
|
+
"trigger": true
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"q": "add a formula to sum the revenue column and total it",
|
|
12
|
+
"trigger": true
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
"q": "chart this raw sales transaction data",
|
|
16
|
+
"trigger": true
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"q": "enter the revenue figures from the company's 10-K into the sheet",
|
|
20
|
+
"trigger": true
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"q": "create a budget spreadsheet with monthly categories",
|
|
24
|
+
"trigger": true
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"q": "what is our LTV to CAC ratio and is it healthy?",
|
|
28
|
+
"trigger": false
|
|
29
|
+
},
|
|
30
|
+
{
|
|
31
|
+
"q": "how long is our cash runway?",
|
|
32
|
+
"trigger": false
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"q": "write a PRD for the new onboarding feature",
|
|
36
|
+
"trigger": false
|
|
37
|
+
},
|
|
38
|
+
{
|
|
39
|
+
"q": "turn these results into a slide deck",
|
|
40
|
+
"trigger": false
|
|
41
|
+
},
|
|
42
|
+
{
|
|
43
|
+
"q": "add a database index on the orders table",
|
|
44
|
+
"trigger": false
|
|
45
|
+
}
|
|
46
|
+
]
|
|
47
|
+
}
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
model_tier: inherit
|
|
3
3
|
name: subagent-orchestration
|
|
4
|
-
description: "Use when orchestrating implementer/judge subagents —
|
|
4
|
+
description: "Use when orchestrating implementer/judge subagents — form gate + eight modes (do-and-judge ±two-stage, steps/parallel/worktrees, competitively, debate, live-app-judge)."
|
|
5
5
|
domain: process
|
|
6
6
|
workspaces:
|
|
7
7
|
- agent-config-maintainer
|
|
@@ -82,11 +82,46 @@ over-scripted prompts that break on first contingency.
|
|
|
82
82
|
- **(d) Pre-declared check-in conditions.** The worker names, at spawn time, the
|
|
83
83
|
conditions under which it will halt and ask ("if login required", "if
|
|
84
84
|
multiple candidates found") — so interrupts are predictable, not surprises.
|
|
85
|
+
- **(e) Attach relevant knowledge — read-only (road-to-opt-subagent-harvest
|
|
86
|
+
P3).** Before dispatch, look up the slice's key identifiers via
|
|
87
|
+
`memory_lookup` / the knowledge cards and attach the top hits to the
|
|
88
|
+
worker prompt AS LEADS — labelled per the source-discovery discipline
|
|
89
|
+
(negative facts + pointers durable; positive structure = hypothesis to
|
|
90
|
+
re-confirm, never a build input). The WRITE half stays forbidden: a
|
|
91
|
+
subagent's output is never auto-persisted into memory — promotion is
|
|
92
|
+
always the human-gated flow (ADR-098 floor). Auto-surface, never
|
|
93
|
+
auto-write.
|
|
85
94
|
|
|
86
95
|
When to delegate at all is [`delegation-policy`](../../rules/delegation-policy.md);
|
|
87
96
|
the spawn boundary is the [`subagent-spawn-contract`](../../contexts/execution/subagent-spawn-contract.md).
|
|
88
97
|
|
|
89
|
-
##
|
|
98
|
+
## Form gate — deterministic, BEFORE mode selection
|
|
99
|
+
|
|
100
|
+
With auto-dispatch on by default (ADR-117), mode selection happens without
|
|
101
|
+
a human in the loop — so the FORM is decided by a static table first, and
|
|
102
|
+
only then is the specific mode picked inside that form. Static table only:
|
|
103
|
+
no learned routing, no self-modifying selector (rejected, stays rejected).
|
|
104
|
+
|
|
105
|
+
| Task shape (structural signal) | Form | Modes in the form |
|
|
106
|
+
|---|---|---|
|
|
107
|
+
| ≥ 2 independent, verifiable slices | parallel | `do-in-parallel`, `do-competitively` |
|
|
108
|
+
| Multi-step cross-wing chain needing filesystem isolation | worktrees | `do-in-worktrees` |
|
|
109
|
+
| Ordered steps with declared dependencies | steps | `do-in-steps` |
|
|
110
|
+
| Single change with non-trivial risk / contested spec / decision | judge | `do-and-judge`, `do-and-judge-two-stage`, `judge-with-debate`, `do-with-live-app-judge` |
|
|
111
|
+
| Single slice below the delegability floor, unstructured, or frontier-priced | none | no dispatch — run in-session |
|
|
112
|
+
|
|
113
|
+
Rules:
|
|
114
|
+
|
|
115
|
+
- The gate consumes the SAME structural signals as
|
|
116
|
+
`auto_dispatch.ts::classifyTask` (slice count, dependency declarations,
|
|
117
|
+
size floor) — it never re-interprets the task text on vibes.
|
|
118
|
+
- Ambiguous shape → `none` (in-session), never a speculative spawn — the
|
|
119
|
+
delegation-policy default.
|
|
120
|
+
- Record the outcome in the telemetry line (`dispatch_mode` field, mode id
|
|
121
|
+
or `none`) so the gate's value is measurable inside the ADR-117
|
|
122
|
+
prove-or-drop window.
|
|
123
|
+
|
|
124
|
+
## The eight modes
|
|
90
125
|
|
|
91
126
|
Each mode has a decision row: when to use, when not, and the expected
|
|
92
127
|
model pairing. Defaults come from
|
|
@@ -178,49 +213,23 @@ migration, public API) where a single judge is too easy to fool.
|
|
|
178
213
|
|
|
179
214
|
### 7. do-in-worktrees
|
|
180
215
|
|
|
181
|
-
Cross-wing or cross-skill chain executed across isolated git
|
|
182
|
-
|
|
183
|
-
the
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
[`finishing-a-development-branch`](../finishing-a-development-branch/SKILL.md).
|
|
216
|
+
Cross-wing or cross-skill chain executed across isolated git worktrees —
|
|
217
|
+
each handoff runs in its own worktree so one step's workspace state never
|
|
218
|
+
leaks into the next. Use for a multi-step cross-wing chain (≥2 senior
|
|
219
|
+
skills, each ≥30 min); not for fast iteration under 30 min (overhead
|
|
220
|
+
dominates). Full handoff shape, example chain, competitive per-candidate
|
|
221
|
+
isolation, and the no-auto-merge Hard Floor →
|
|
222
|
+
[`subagent-modes-detail`](../../agent-src/contexts/execution/subagent-modes-detail.md) § Mode 7.
|
|
189
223
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
**Example chain (W3 launch):** `positioning-strategy` (worktree A) →
|
|
201
|
-
`messaging-architecture` (worktree B, consumes positioning's
|
|
202
|
-
`positioning-statement.md`) → `gtm-launch` (worktree C, consumes
|
|
203
|
-
both prior artifacts). Each worktree carries one branch; the chain
|
|
204
|
-
end produces a single integration PR.
|
|
205
|
-
|
|
206
|
-
**Anti-pattern:** do not use for fast iteration loops where each
|
|
207
|
-
step is under ~30 minutes. The branch-creation, context-switch, and
|
|
208
|
-
worktree-cleanup cost dominates. Stick with mode 1 (do-and-judge)
|
|
209
|
-
or mode 3 (do-in-steps) for those.
|
|
210
|
-
|
|
211
|
-
**Competitive variant — per-candidate isolation.** When mode 5
|
|
212
|
-
(`do-competitively`) is combined with worktrees, each candidate
|
|
213
|
-
implementer runs in its own worktree (so candidates cannot read each
|
|
214
|
-
other's open files or branch state). Selection rules:
|
|
215
|
-
|
|
216
|
-
- **No auto-merge.** The orchestrator never merges a candidate
|
|
217
|
-
branch. Hard Floor per [`non-destructive-by-default`](../../rules/non-destructive-by-default.md) —
|
|
218
|
-
applies even under standing autonomy. ADR-005 records the reasoning.
|
|
219
|
-
- **Ranked presentation.** Judge ranks candidates (1..N) with a
|
|
220
|
-
one-line justification per rank; user picks the winner.
|
|
221
|
-
- **Loser worktrees stay.** The orchestrator does not delete losing
|
|
222
|
-
worktrees automatically — the user keeps the option to harvest a
|
|
223
|
-
partial idea before cleanup.
|
|
224
|
+
### 8. do-with-live-app-judge (gated — UI-heavy tasks)
|
|
225
|
+
|
|
226
|
+
Implementer ships the change AND starts the dev server; the judge drives
|
|
227
|
+
the RUNNING app (Playwright / browser) against a written rubric, never
|
|
228
|
+
reading the diff. Use for UI-heavy change where "looks right in the diff"
|
|
229
|
+
≠ "works in the app"; not for backend/logic (a diff judge is cheaper).
|
|
230
|
+
Experimental until `verdict_changed_outcome` telemetry proves it. Rubric,
|
|
231
|
+
adoption gate, and the async-verifier future candidate →
|
|
232
|
+
[`subagent-modes-detail`](../../agent-src/contexts/execution/subagent-modes-detail.md) § Mode 8.
|
|
224
233
|
|
|
225
234
|
## Status taxonomy — every subagent return uses one envelope
|
|
226
235
|
|
|
@@ -256,7 +265,7 @@ Each mode's literal dispatch template lives under
|
|
|
256
265
|
matching prompt at dispatch time and substitutes `{{placeholders}}`.
|
|
257
266
|
Edits to a prompt do not bloat this skill against the 400-line sunset
|
|
258
267
|
trigger; `tests/test_subagent_prompt_loading.py` confirms each of the
|
|
259
|
-
|
|
268
|
+
eight modes resolves to a loadable prompt that cites all four taxonomy
|
|
260
269
|
statuses.
|
|
261
270
|
|
|
262
271
|
## Procedure
|
|
@@ -285,7 +294,7 @@ same context, **stop** and report. Do not improvise.
|
|
|
285
294
|
|
|
286
295
|
### 3. Pick the mode
|
|
287
296
|
|
|
288
|
-
|
|
297
|
+
Run the form gate first, then match task shape to one of the eight modes. When two modes could fit,
|
|
289
298
|
prefer the cheaper one (`do-and-judge` < `do-and-judge-two-stage` <
|
|
290
299
|
`do-in-steps` < `do-in-parallel` < `do-competitively` <
|
|
291
300
|
`judge-with-debate` < `do-in-worktrees`).
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "subagent-orchestration",
|
|
3
|
+
"_calibration": "Golden E2E task 2 (road-to-fable-feedback-5 Phase 4c; council 2026-07-10 ranked bad-lite-output delegation second-highest signal/cost). Exercises the never-adopt-unverified Iron Law (delegation-policy) + the escalation path + the telemetry pairing (first_pass_success/escalated per orchestration-telemetry.md § Operationalization). Deterministic `contains` assertions pin the telemetry vocabulary; rubric pins the verification behavior. Baseline: authored 2026-07-10, pending first host-scored run.",
|
|
4
|
+
"scenarios": [
|
|
5
|
+
{
|
|
6
|
+
"id": "sao-bad-lite-output-escalation",
|
|
7
|
+
"prompt": "You are orchestrating: a lite-tier subagent was dispatched to implement `slugify(title: string): string` (lowercase, spaces to hyphens, strip non-alphanumerics, collapse repeated hyphens). Its return is: `export function slugify(title: string) { return title.toLowerCase().replace(' ', '-'); }` with no tests. Process this return per the orchestration contract and record the dispatch telemetry.",
|
|
8
|
+
"assertions": [
|
|
9
|
+
{
|
|
10
|
+
"kind": "contains",
|
|
11
|
+
"value": "first_pass_success"
|
|
12
|
+
},
|
|
13
|
+
{
|
|
14
|
+
"kind": "contains",
|
|
15
|
+
"value": "escalated"
|
|
16
|
+
},
|
|
17
|
+
{
|
|
18
|
+
"kind": "rubric",
|
|
19
|
+
"criterion": "The return is NOT adopted unverified: verification runs against the stated spec and catches at least two planted defects (String.replace with a string only replaces the FIRST space; non-alphanumerics are not stripped; repeated hyphens are not collapsed; no tests). The orchestrator never merges the defective return as-is."
|
|
20
|
+
},
|
|
21
|
+
{
|
|
22
|
+
"kind": "rubric",
|
|
23
|
+
"criterion": "After the verification failure the slice is re-dispatched on a higher tier (or corrected with the failure fed back), and the telemetry records escalated=true and first_pass_success=false per the operationalization table (business-logic rework counts as rework; formatting would not). The final adopted output satisfies the full spec."
|
|
24
|
+
}
|
|
25
|
+
]
|
|
26
|
+
},
|
|
27
|
+
{
|
|
28
|
+
"id": "sao-good-return-no-spurious-escalation",
|
|
29
|
+
"prompt": "You are orchestrating: a lite-tier subagent was dispatched to rename the constant `MAX_RETRIES` to `MAX_ATTEMPTS` across a 3-file module. Its return is a diff renaming all 7 occurrences, all references updated, tests still green. Process this return per the orchestration contract and record the dispatch telemetry.",
|
|
30
|
+
"assertions": [
|
|
31
|
+
{
|
|
32
|
+
"kind": "contains",
|
|
33
|
+
"value": "first_pass_success"
|
|
34
|
+
},
|
|
35
|
+
{
|
|
36
|
+
"kind": "rubric",
|
|
37
|
+
"criterion": "Negative control: the clean return is verified and adopted WITHOUT rework or escalation — telemetry records first_pass_success=true and escalated=false. No spurious findings are invented to justify a re-dispatch."
|
|
38
|
+
}
|
|
39
|
+
]
|
|
40
|
+
}
|
|
41
|
+
]
|
|
42
|
+
}
|
|
@@ -185,6 +185,16 @@ Pairs with [`context-hygiene § Read-Loop Detection`](../../rules/context-hygien
|
|
|
185
185
|
6. **Full suite at the end, not between laps.** It is the gate, not
|
|
186
186
|
the feedback loop.
|
|
187
187
|
|
|
188
|
+
**Failure-signature triage — same taxonomy as [`context-hygiene § The 3-Failure Rule`](../../rules/context-hygiene.md#the-3-failure-rule).**
|
|
189
|
+
The **failure signature** = **same target + same error class** (same failing test
|
|
190
|
+
+ same assertion, same lint rule id, same build error). **Same failure signature
|
|
191
|
+
twice → stop and pivot** — don't spend the next lap on a near-identical retry; a
|
|
192
|
+
repeated identical signature means the hypothesis is wrong, not under-applied. A
|
|
193
|
+
**new error signature each attempt = progress** and the counter continues. The
|
|
194
|
+
**hard-blocker classes** (missing credentials, permission denied, spend/quota/rate
|
|
195
|
+
limit, external-service `5xx`) skip retries entirely — surface on first
|
|
196
|
+
occurrence; another lap cannot fix them.
|
|
197
|
+
|
|
188
198
|
**Anti-patterns this loop prevents:**
|
|
189
199
|
|
|
190
200
|
* Editing source to make tests pass when the test was wrong (or
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "systematic-debugging",
|
|
3
|
+
"_calibration": "Behavioural fixture (road-to-skill-eval-coverage Phase 2, default-surface). Deterministic assertions on a real output property where reliable; rubric pins the qualitative core as a known-limit (pass ratified in PR review, not a hidden judge). finding_floor n is a conservative floor, not a calibrated per-host number.",
|
|
4
|
+
"scenarios": [
|
|
5
|
+
{
|
|
6
|
+
"id": "debug-flaky-test",
|
|
7
|
+
"prompt": "A test passes locally but fails in CI intermittently. Walk me through debugging it.",
|
|
8
|
+
"assertions": [
|
|
9
|
+
{
|
|
10
|
+
"kind": "contains",
|
|
11
|
+
"value": "hypothesis"
|
|
12
|
+
},
|
|
13
|
+
{
|
|
14
|
+
"kind": "rubric",
|
|
15
|
+
"criterion": "Forms a hypothesis, isolates one variable, and runs one-test-one-fix-one-rerun rather than shotgun edits."
|
|
16
|
+
}
|
|
17
|
+
]
|
|
18
|
+
}
|
|
19
|
+
]
|
|
20
|
+
}
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
---
|
|
2
|
+
model_tier: medium
|
|
3
|
+
name: test-case-discovery
|
|
4
|
+
description: "Use BEFORE writing any test — enumerate cases per behavior (happy / boundary / error / abuse), prioritize by likelihood × impact, cross-check via subagent — even if the user just says 'add tests'."
|
|
5
|
+
domain: quality
|
|
6
|
+
workspaces:
|
|
7
|
+
- engineering
|
|
8
|
+
packs:
|
|
9
|
+
- engineering-base
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
# test-case-discovery
|
|
13
|
+
|
|
14
|
+
AI-written test suites default to **too few tests, mostly happy path**. This skill is the discovery step that runs *before* the first test is written: enumerate what could break, pick the most important cases, then hand the list to [`test-driven-development`](../test-driven-development/SKILL.md) or the stack skill. The goal is **breadth AND quality** — happy path AND boundaries AND error paths — with a hard cap so the output is never a pile of trivial tests.
|
|
15
|
+
|
|
16
|
+
This is the **before** layer. Assertion quality is [`testing-anti-patterns`](../testing-anti-patterns/SKILL.md); after-the-fact coverage review is [`judge-test-coverage`](../judge-test-coverage/SKILL.md).
|
|
17
|
+
|
|
18
|
+
## When to use
|
|
19
|
+
|
|
20
|
+
- About to write tests for new or changed behavior — the `test-driven-development` RED phase and `/tests create` route here as a mandatory first step.
|
|
21
|
+
- A test task arrives as "add tests" / "write tests for this branch" with no case list.
|
|
22
|
+
- Reviewing a plan whose testing section only names the happy path.
|
|
23
|
+
|
|
24
|
+
Do NOT use when:
|
|
25
|
+
|
|
26
|
+
- The diff is trivial (< 10 lines, pure rename/refactor, no new behavior) — write 1 happy + 1 boundary case and move on; no matrix.
|
|
27
|
+
- The problem is assertion quality (overfit, tautology, mock-the-mock) → [`testing-anti-patterns`](../testing-anti-patterns/SKILL.md).
|
|
28
|
+
- A test is failing for non-obvious reasons → [`systematic-debugging`](../systematic-debugging/SKILL.md).
|
|
29
|
+
- The tests already exist and the question is "are they enough?" → [`judge-test-coverage`](../judge-test-coverage/SKILL.md).
|
|
30
|
+
|
|
31
|
+
## The Iron Law
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
NO TEST IS WRITTEN BEFORE THE CASE LIST EXISTS.
|
|
35
|
+
ENUMERATE → PRIORITIZE → THEN WRITE. NEVER HAPPY-PATH-ONLY.
|
|
36
|
+
FLOOR PER BEHAVIOR: 1 HAPPY + 1 BOUNDARY + 1 ERROR (+1 ABUSE ON SECURITY PATHS).
|
|
37
|
+
CAP PER BEHAVIOR: 5–8 CASES — EACH MUST FAIL FOR A DISTINCT REASON.
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Procedure — the five-step funnel
|
|
41
|
+
|
|
42
|
+
### 1. Behavior inventory (30 seconds)
|
|
43
|
+
|
|
44
|
+
List the 1–3 **distinct observable behaviors** the change introduces or modifies — one sentence each: *"When X happens, the system should do Y."* Each behavior gets its own case list. If you cannot state a behavior in one sentence, split it.
|
|
45
|
+
|
|
46
|
+
### 2. Dimension scan (per behavior)
|
|
47
|
+
|
|
48
|
+
Walk the dimensions below; mark **3–5 as HIGH** (most likely to occur × highest impact when wrong). Skip dimensions the behavior structurally cannot hit — but say so, don't silently omit.
|
|
49
|
+
|
|
50
|
+
- **Input validity** — null/undefined, empty string/array, zero, negative, max bounds/overflow, type mismatch, malformed format, Unicode/whitespace, off-by-one.
|
|
51
|
+
- **State** — missing prerequisites, invalid state transitions, idempotency/double-submit, duplicates and ordering.
|
|
52
|
+
- **Collaborator failure** — timeout, 4xx/5xx, partial or malformed response, thrown exceptions, empty result sets.
|
|
53
|
+
- **Concurrency & time** — races, retries, timezone/DST, clock skew (only when the surface touches them).
|
|
54
|
+
- **Security surface** (authz / tenant / data access) — missing/expired credentials, not-owner, cross-tenant, privilege escalation, injection payload. On data-returning endpoints the three negative tests from `broken-access-control` (unauthenticated → 401, not-owner → 403/404, cross-tenant → 403/404) are the floor, not optional extras.
|
|
55
|
+
|
|
56
|
+
### 3. Case synthesis
|
|
57
|
+
|
|
58
|
+
Generate ≥ 1 concrete case per HIGH dimension, plus the happy path. Enforce the floor: **1 happy + 1 boundary + 1 error per behavior, +1 abuse case when the behavior is security-relevant.** Name each case like a test (`it rejects an expired token`), not like a category ("error handling").
|
|
59
|
+
|
|
60
|
+
### 4. Subagent cross-check (gated, with mandatory fallback)
|
|
61
|
+
|
|
62
|
+
Widen the list with a second pair of eyes:
|
|
63
|
+
|
|
64
|
+
- **Subagent path** — when the host supports subagent spawn and subagents are enabled (`subagents.enabled`, `subagents.auto` ≠ off): spawn **one** subagent with an adversarial lens — *"Behavior: {sentence}. Case list so far: {list}. As a security researcher / SRE / chaos engineer: which non-obvious failure mode or abuse vector is missing?"* On security-critical surfaces a **second** subagent with a different lens is allowed; more is waste. Merge only novel cases.
|
|
65
|
+
- **Degraded path (mandatory when no subagent runs)** — a self-review pass with an explicit lens shift: re-walk step 2 asking *"what breaks in production? what would an attacker try?"* and emit the same output shape (1–2 additional cases, or explicitly "none found").
|
|
66
|
+
|
|
67
|
+
Unavailability of subagents never skips this step — it only switches the path.
|
|
68
|
+
|
|
69
|
+
### 5. Prioritize and cap
|
|
70
|
+
|
|
71
|
+
If a behavior collected more than 8 cases: rank by likelihood × blast radius, keep the top 5–8. Quality gate: **each surviving case must be able to fail for a distinct reason** (different code path / exception / assertion). The same assertion fed two interchangeable input values is ONE case — use a data provider, not a second test. Record dropped cases in one line each so the discovery work is not lost.
|
|
72
|
+
|
|
73
|
+
Then hand the list to [`test-driven-development`](../test-driven-development/SKILL.md) — every case runs its own RED → GREEN cycle — or to the stack skill ([`pest-testing`](../pest-testing/SKILL.md), [`api-testing`](../api-testing/SKILL.md), [`playwright-testing`](../playwright-testing/SKILL.md)).
|
|
74
|
+
|
|
75
|
+
## Output format
|
|
76
|
+
|
|
77
|
+
1. The **case matrix** per behavior — case name → category (happy / boundary / error / abuse) → one-line why — in the reply or as a comment block atop the test file.
|
|
78
|
+
2. The **floor check**, stated explicitly: `floor met: happy ✓ boundary ✓ error ✓ (abuse: n/a — no security surface)`.
|
|
79
|
+
3. The **cross-check result**: which path ran (subagent + lens, or degraded self-review) and which cases it added — or explicitly "none".
|
|
80
|
+
4. **Dropped/deferred cases** with a one-line reason each.
|
|
81
|
+
|
|
82
|
+
## Gotcha
|
|
83
|
+
|
|
84
|
+
- Line coverage lies — 100 % of lines with happy-path inputs still misses every boundary. Count **cases**, not lines.
|
|
85
|
+
- The floor is per **behavior**, not per file — one boundary test in a 5-behavior file does not clear four other behaviors.
|
|
86
|
+
- Enumeration is not an implementation license: the cap and [`testing-anti-patterns`](../testing-anti-patterns/SKILL.md) Anti-Pattern 6 still apply — 6 meaningful tests beat 20 trivial ones.
|
|
87
|
+
- "No subagent available" is not "skip step 4" — the degraded self-review is mandatory and cheap.
|
|
88
|
+
- A case list written *after* the tests is decoration — the funnel runs before the first RED.
|
|
89
|
+
|
|
90
|
+
## Do NOT
|
|
91
|
+
|
|
92
|
+
- Do NOT write the first test before the case list exists.
|
|
93
|
+
- Do NOT stop after the happy-path case — the floor is 1 happy + 1 boundary + 1 error per behavior.
|
|
94
|
+
- Do NOT pad the list with cases that fail for the same reason — data providers cover value variations.
|
|
95
|
+
- Do NOT skip the step-4 cross-check because subagents are unavailable — run the degraded self-review.
|
|
96
|
+
- Do NOT run the full funnel on trivial diffs — 1 happy + 1 boundary suffices there.
|
|
97
|
+
- Do NOT drop enumerated cases silently — record them with a reason.
|
|
98
|
+
|
|
99
|
+
## Auto-trigger keywords
|
|
100
|
+
|
|
101
|
+
- test case discovery
|
|
102
|
+
- what to test
|
|
103
|
+
- test coverage plan
|
|
104
|
+
- edge cases
|
|
105
|
+
- boundary cases
|
|
106
|
+
- happy path only
|
|
107
|
+
- broad test coverage
|
|
108
|
+
- case matrix
|
|
109
|
+
- negative tests
|
|
110
|
+
|
|
111
|
+
## See also
|
|
112
|
+
|
|
113
|
+
- [`test-driven-development`](../test-driven-development/SKILL.md) — consumes the case list; one RED → GREEN cycle per case.
|
|
114
|
+
- [`testing-anti-patterns`](../testing-anti-patterns/SKILL.md) — assertion-quality floor (overfit, tautology, gaming the green).
|
|
115
|
+
- [`judge-test-coverage`](../judge-test-coverage/SKILL.md) — after-the-fact coverage review.
|
|
116
|
+
- [`pest-testing`](../pest-testing/SKILL.md) · [`api-testing`](../api-testing/SKILL.md) · [`playwright-testing`](../playwright-testing/SKILL.md) — stack-specific execution.
|
|
117
|
+
- Rule `senior-engineering-discipline` — the Iron-Law floor this procedure operationalizes.
|
|
118
|
+
- Rule `broken-access-control` — the three negative tests on data-returning endpoints.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "test-case-discovery",
|
|
3
|
+
"last_eval": "2026-07-10",
|
|
4
|
+
"description": "5 should-trigger + 5 should-not-trigger queries. Should-trigger covers DE + EN phrasings for pre-write test-case enumeration (what to test, edge cases, coverage breadth). Should-not-trigger covers near-miss neighbours: assertion quality (testing-anti-patterns), after-the-fact coverage review (judge-test-coverage), failing-test diagnosis (systematic-debugging), stack-specific execution.",
|
|
5
|
+
"queries": [
|
|
6
|
+
{
|
|
7
|
+
"q": "what should we test here — I don't want happy-path-only coverage",
|
|
8
|
+
"trigger": true
|
|
9
|
+
},
|
|
10
|
+
{
|
|
11
|
+
"q": "enumerate the edge cases for this endpoint before we write the tests",
|
|
12
|
+
"trigger": true
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
"q": "welche Testfälle brauchen wir für dieses Feature — auch Grenzfälle und Fehlerpfade?",
|
|
16
|
+
"trigger": true
|
|
17
|
+
},
|
|
18
|
+
{
|
|
19
|
+
"q": "build a case matrix for this behavior — happy, boundary, error, abuse",
|
|
20
|
+
"trigger": true
|
|
21
|
+
},
|
|
22
|
+
{
|
|
23
|
+
"q": "ask a subagent what else we could test so coverage is broad",
|
|
24
|
+
"trigger": true
|
|
25
|
+
},
|
|
26
|
+
{
|
|
27
|
+
"q": "this assertion looks tautological — is the test overfit?",
|
|
28
|
+
"trigger": false,
|
|
29
|
+
"note": "assertion quality → testing-anti-patterns"
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
"q": "are the tests on this finished diff enough?",
|
|
33
|
+
"trigger": false,
|
|
34
|
+
"note": "after-the-fact coverage review → judge-test-coverage"
|
|
35
|
+
},
|
|
36
|
+
{
|
|
37
|
+
"q": "this Pest test keeps failing and I don't understand why",
|
|
38
|
+
"trigger": false,
|
|
39
|
+
"note": "failing-test diagnosis → systematic-debugging"
|
|
40
|
+
},
|
|
41
|
+
{
|
|
42
|
+
"q": "how do I mock Http in a Laravel feature test?",
|
|
43
|
+
"trigger": false,
|
|
44
|
+
"note": "stack-specific execution → pest-testing"
|
|
45
|
+
},
|
|
46
|
+
{
|
|
47
|
+
"q": "schreib die Playwright-Specs für den Login-Flow, die Fälle stehen schon im Ticket",
|
|
48
|
+
"trigger": false,
|
|
49
|
+
"note": "case list already exists → playwright-testing executes it"
|
|
50
|
+
}
|
|
51
|
+
]
|
|
52
|
+
}
|
|
@@ -92,7 +92,25 @@ State in one sentence: *"When X happens, the system should do Y."*
|
|
|
92
92
|
If you cannot state it in one sentence, the scope is too big — split into
|
|
93
93
|
multiple tests, each covering one sentence.
|
|
94
94
|
|
|
95
|
-
### 2.
|
|
95
|
+
### 2. Enumerate the cases — discovery before writing (MANDATORY)
|
|
96
|
+
|
|
97
|
+
Before the first test is written, run the
|
|
98
|
+
[`test-case-discovery`](../test-case-discovery/SKILL.md) funnel for the
|
|
99
|
+
behavior: dimension scan → case synthesis → optional subagent cross-check →
|
|
100
|
+
prioritization. Do not proceed to step 3 with fewer than the floor:
|
|
101
|
+
|
|
102
|
+
* **1 happy + 1 boundary + 1 error case per behavior** (+1 abuse case on
|
|
103
|
+
security-relevant paths).
|
|
104
|
+
* Cap at 5–8 cases per behavior; each must be able to fail for a
|
|
105
|
+
**distinct** reason.
|
|
106
|
+
* Trivial change (< 10 lines, pure refactor, no new behavior) → skip the
|
|
107
|
+
funnel; 1 happy + 1 boundary case suffices.
|
|
108
|
+
|
|
109
|
+
Each case from the list then gets its own RED → GREEN cycle (steps 3–6).
|
|
110
|
+
A behavior whose only test is the happy path is **not done** — it is the
|
|
111
|
+
first item of an unfinished case list.
|
|
112
|
+
|
|
113
|
+
### 3. Write the failing test first
|
|
96
114
|
|
|
97
115
|
Write the smallest test that expresses the sentence from step 1.
|
|
98
116
|
|
|
@@ -101,7 +119,7 @@ Write the smallest test that expresses the sentence from step 1.
|
|
|
101
119
|
* Real code paths, not mocks — mock only at I/O boundaries (HTTP, DB, time).
|
|
102
120
|
* Use a descriptive name: `it_rejects_empty_email`, not `test_email_1`.
|
|
103
121
|
|
|
104
|
-
###
|
|
122
|
+
### 4. Run the test and watch it fail
|
|
105
123
|
|
|
106
124
|
Execute the single test (targeted, not the full suite):
|
|
107
125
|
|
|
@@ -120,7 +138,7 @@ Required observations **before proceeding**:
|
|
|
120
138
|
* If the test passes immediately → it does not test what you think. Fix
|
|
121
139
|
the test, do not start writing production code.
|
|
122
140
|
|
|
123
|
-
###
|
|
141
|
+
### 5. Write minimum code to pass
|
|
124
142
|
|
|
125
143
|
Add **just enough** production code to make the test green. No extra
|
|
126
144
|
features, no unrelated refactoring, no "while I'm here" cleanups.
|
|
@@ -129,7 +147,7 @@ If you feel the urge to add a parameter, edge case, or helper not covered
|
|
|
129
147
|
by the current test — stop. That belongs in the next RED step, not this
|
|
130
148
|
GREEN step.
|
|
131
149
|
|
|
132
|
-
###
|
|
150
|
+
### 6. Run again and watch it pass
|
|
133
151
|
|
|
134
152
|
Re-run the same targeted command. Required:
|
|
135
153
|
|
|
@@ -137,7 +155,7 @@ Re-run the same targeted command. Required:
|
|
|
137
155
|
* No previously green tests have turned red.
|
|
138
156
|
* Test output is clean (no new warnings, deprecations, or noise).
|
|
139
157
|
|
|
140
|
-
###
|
|
158
|
+
### 7. Refactor (only if green)
|
|
141
159
|
|
|
142
160
|
With all tests green, you may:
|
|
143
161
|
|
|
@@ -148,7 +166,7 @@ With all tests green, you may:
|
|
|
148
166
|
Do **not** add new behavior during refactor — that needs its own failing
|
|
149
167
|
test first. Re-run tests after the refactor to confirm still-green.
|
|
150
168
|
|
|
151
|
-
###
|
|
169
|
+
### 8. Repeat for the next behavior
|
|
152
170
|
|
|
153
171
|
Back to step 1 with the next single-sentence behavior.
|
|
154
172
|
|
|
@@ -260,6 +278,8 @@ wait for their own failing tests.
|
|
|
260
278
|
|
|
261
279
|
* Do NOT write or modify production code before the failing test exists
|
|
262
280
|
and has been observed to fail
|
|
281
|
+
* Do NOT stop after the happy-path test — work through the enumerated
|
|
282
|
+
case list from step 2 (boundary, error, abuse where relevant)
|
|
263
283
|
* Do NOT accept a test that never failed as evidence the code works
|
|
264
284
|
* Do NOT bundle refactors into the GREEN step
|
|
265
285
|
* Do NOT silence a flaky test — diagnose it, or delete it
|
|
@@ -274,6 +294,8 @@ wait for their own failing tests.
|
|
|
274
294
|
|
|
275
295
|
## When to hand over to another skill
|
|
276
296
|
|
|
297
|
+
* Enumerating what to test before writing (case matrix, dimension scan,
|
|
298
|
+
subagent cross-check) → [`test-case-discovery`](../test-case-discovery/SKILL.md)
|
|
277
299
|
* Project type-checker / linter / formatter (PHPStan, ECS, Rector for PHP — tsc / eslint / prettier for TS — ruff / mypy for Python) → [`quality-tools`](../quality-tools/SKILL.md)
|
|
278
300
|
* Full Pest conventions and Laravel test helpers → [`pest-testing`](../pest-testing/SKILL.md)
|
|
279
301
|
* Running tests inside Docker → [`tests-execute`](../tests-execute/SKILL.md)
|
|
@@ -285,6 +307,10 @@ wait for their own failing tests.
|
|
|
285
307
|
Before marking TDD work complete:
|
|
286
308
|
|
|
287
309
|
* [ ] Every new behavior has a test
|
|
310
|
+
* [ ] The case list from step 2 exists and meets the floor (1 happy +
|
|
311
|
+
1 boundary + 1 error per behavior; abuse case on security paths)
|
|
312
|
+
* [ ] Every enumerated case is either tested or recorded as dropped with
|
|
313
|
+
a one-line reason — no behavior ships happy-path-only
|
|
288
314
|
* [ ] Each test was observed to fail first, with a matching failure message
|
|
289
315
|
* [ ] The minimum code was written to turn each RED into GREEN
|
|
290
316
|
* [ ] All targeted tests pass
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "test-driven-development",
|
|
3
|
+
"_calibration": "Behavioural fixture (road-to-skill-eval-coverage Phase 2, default-surface). Deterministic assertions on a real output property where reliable; rubric pins the qualitative core as a known-limit (pass ratified in PR review, not a hidden judge). finding_floor n is a conservative floor, not a calibrated per-host number.",
|
|
4
|
+
"scenarios": [
|
|
5
|
+
{
|
|
6
|
+
"id": "tdd-new-validator",
|
|
7
|
+
"prompt": "TDD a password-strength validator (min length, mixed case, a digit).",
|
|
8
|
+
"assertions": [
|
|
9
|
+
{
|
|
10
|
+
"kind": "rubric",
|
|
11
|
+
"criterion": "Writes a failing test FIRST (red), then the minimal code (green), then refactors — and covers boundary + negative cases, not only the happy path."
|
|
12
|
+
}
|
|
13
|
+
]
|
|
14
|
+
}
|
|
15
|
+
]
|
|
16
|
+
}
|
|
@@ -31,6 +31,7 @@ still test-after-the-fact.
|
|
|
31
31
|
Do NOT use when:
|
|
32
32
|
|
|
33
33
|
- You need to *write* tests (no anti-pattern present yet) — route to [`pest-testing`](../pest-testing/SKILL.md) or [`test-driven-development`](../test-driven-development/SKILL.md).
|
|
34
|
+
- You need to *enumerate* what to test before writing (case matrix, edge-case discovery) — route to [`test-case-discovery`](../test-case-discovery/SKILL.md).
|
|
34
35
|
- The test failure is a real bug — route to [`systematic-debugging`](../systematic-debugging/SKILL.md).
|
|
35
36
|
- You need overall coverage assessment of a finished diff — route to [`judge-test-coverage`](../judge-test-coverage/SKILL.md).
|
|
36
37
|
|
|
@@ -130,7 +131,7 @@ Gate: a feature is not complete until a failing-then-passing test cycle ran for
|
|
|
130
131
|
|
|
131
132
|
### Anti-Pattern 6 — Overfit / tautological assertions & unrealistic test data
|
|
132
133
|
|
|
133
|
-
Symptom: the test passes only for one crafted input and proves nothing about the general rule — a hardcoded expected value the code always
|
|
134
|
+
Symptom: the test passes only for one crafted input and proves nothing about the general rule — a hardcoded expected value the code will always emit, a narrow regex pinned to a fixed date/id, or an assertion that restates the seed. AI-written tests reach the same coverage as human tests but ~4× worse fault detection (the weak-oracle problem): they execute the path and then assert something too weak to catch a bug. Same shape as mocking-the-mock — asserting on your own construction rather than the system's behavior.
|
|
134
135
|
|
|
135
136
|
Gate:
|
|
136
137
|
|
|
@@ -141,7 +142,7 @@ BEFORE writing an assertion:
|
|
|
141
142
|
STOP — derive the expected from the input, and add cases that vary it.
|
|
142
143
|
```
|
|
143
144
|
|
|
144
|
-
Test-data realism — seeders/factories emit random data; the test derives its expectation from that data, never a hardcoded literal:
|
|
145
|
+
Test-data realism — seeders/factories emit random data; the test derives its expectation from that data, never from a hardcoded literal:
|
|
145
146
|
|
|
146
147
|
```js
|
|
147
148
|
// ❌ WRONG — hardcoded expectation, single happy case
|
|
@@ -160,11 +161,11 @@ expect((await api.get('/users/abc')).status).toBe(400) // error: invalid
|
|
|
160
161
|
expect([403, 404]).toContain((await api.get(`/users/${otherTenantUser.id}`)).status) // abuse: tenant isolation (404 hides existence)
|
|
161
162
|
```
|
|
162
163
|
|
|
163
|
-
Deriving from seeded data is necessary but **not sufficient** — a test that only reads back the exact fields it seeded is still overfit (a code
|
|
164
|
+
Deriving from seeded data is necessary but **not sufficient** — a test that only reads back the exact fields it seeded is still overfit (a mutation in the code survives it). A real test also exercises boundary (empty / null / max / Unicode), error (missing / invalid), and, on security-sensitive paths, an abuse case (IDOR, injection string, XSS payload). Coverage of *cases*, not just lines.
|
|
164
165
|
|
|
165
166
|
### Anti-Pattern 7 — Gaming the green (test-integrity)
|
|
166
167
|
|
|
167
|
-
Symptom: a test fails, and the "fix" edits the *test* instead of the code — `.skip` / `it.skip` / `xfail` / `@Disabled` on the failing case, deleting the failing assertion, loosening `toBe(x)` to `toBeTruthy()`, or rewriting `expected` to whatever the (possibly buggy) code now emits. The suite goes green while the behavior stays broken — worst outcome, because green now *hides* the bug.
|
|
168
|
+
Symptom: a test fails, and the "fix" edits the *test* instead of the code — `.skip` / `it.skip` / `xfail` / `@Disabled` on the failing case, deleting the failing assertion, loosening `toBe(x)` to `toBeTruthy()`, or rewriting the `expected` value to whatever the (possibly buggy) code now emits. The suite goes green while the behavior stays broken — the worst outcome, because green now *hides* the bug. This is the automation-bias trap: under pressure to finish, the agent optimizes the signal (green) instead of the target (correct behavior).
|
|
168
169
|
|
|
169
170
|
Gate:
|
|
170
171
|
|
|
@@ -184,7 +185,7 @@ Backstop grep — skip/xfail added to chase green (a hit in a diff that also "fi
|
|
|
184
185
|
rg -n '\b(it|test|describe)\.skip\b|\.only\b|xit\(|xdescribe\(|@pytest\.mark\.(skip|xfail)|@Disabled|->markTestSkipped\(' .
|
|
185
186
|
```
|
|
186
187
|
|
|
187
|
-
|
|
188
|
+
This is the test-surface instance of the `autonomous-execution` N=3 / allowlist-growth antipattern (bulk `skip`/`xfail` to force green counts as the tool being wrong, not the content) and the `verify-before-complete` floor (green must be *earned*, not manufactured).
|
|
188
189
|
|
|
189
190
|
## Output format
|
|
190
191
|
|
|
@@ -227,7 +228,7 @@ Test-surface instance of the `autonomous-execution` N=3 / allowlist-growth antip
|
|
|
227
228
|
|
|
228
229
|
## Provenance
|
|
229
230
|
|
|
230
|
-
- Adopted from: an external reference (
|
|
231
|
+
- Adopted from: an external reference (internal provenance, redacted).
|
|
231
232
|
- Cross-linked: [`pest-testing`](../pest-testing/SKILL.md), [`test-driven-development`](../test-driven-development/SKILL.md), [`judge-test-coverage`](../judge-test-coverage/SKILL.md).
|
|
232
233
|
- Provenance registry: `agents/settings/contexts/skills-provenance.yml` (entry: `testing-anti-patterns`).
|
|
233
234
|
- Iron-Law floor: `verify-before-complete`, `skill-quality`, `senior-engineering-discipline`.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
{
|
|
2
|
+
"skill": "threat-modeling",
|
|
3
|
+
"_calibration": "Behavioural fixture (road-to-skill-eval-coverage Phase 2, default-surface). Deterministic assertions on a real output property where reliable; rubric pins the qualitative core as a known-limit (pass ratified in PR review, not a hidden judge). finding_floor n is a conservative floor, not a calibrated per-host number.",
|
|
4
|
+
"scenarios": [
|
|
5
|
+
{
|
|
6
|
+
"id": "threat-model-upload",
|
|
7
|
+
"prompt": "Threat-model a new file-upload endpoint that accepts user files.",
|
|
8
|
+
"assertions": [
|
|
9
|
+
{
|
|
10
|
+
"kind": "finding_floor",
|
|
11
|
+
"n": 3,
|
|
12
|
+
"pattern": "(?m)^\\s*[-*+]\\s+"
|
|
13
|
+
},
|
|
14
|
+
{
|
|
15
|
+
"kind": "rubric",
|
|
16
|
+
"criterion": "Enumerates concrete abuse cases (path traversal, SSRF via URL fetch, malicious file, authz bypass) and the missing control per case — not generic 'validate input'."
|
|
17
|
+
}
|
|
18
|
+
]
|
|
19
|
+
}
|
|
20
|
+
]
|
|
21
|
+
}
|