harness-everything 0.2.2 → 0.4.0

package/evals/environment-detection/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-environment-detection-001
+name: Off-scope prompt must not trigger environment-detection
+description: A request outside environment-detection's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Write a short haiku about autumn rain."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "Preflight"
+        - "package manager"

package/evals/environment-detection/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-environment-detection-001
+name: On-scope prompt triggers environment-detection
+description: "Use at session start (Discover phase) to detect and align with the operating system, shell (Git Bash vs PowerShell vs CMD), package managers, and dev tools; prevents blind command execution and repetitive tool errors."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Run environment detection so you know my OS, shell, and package manager before touching anything."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "shell"

package/evals/eval-harness/eval.yaml

@@ -0,0 +1,26 @@
+name: eval-harness-routing-eval
+description: Trigger precision eval for the eval-harness skill.
+skill: eval-harness
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did eval-harness trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/eval-harness/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-eval-harness-001
+name: Off-scope prompt must not trigger eval-harness
+description: A request outside eval-harness's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Refactor the auth module to use middleware."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "eval-harness"
+        - "scoring rubric"

package/evals/eval-harness/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-eval-harness-001
+name: On-scope prompt triggers eval-harness
+description: "Evaluate AI agent performance across correctness, token efficiency, anti-loop focus, and environment awareness; save a 4-dimension scorecard. Use when asked to run a benchmark or score an execution log or conversation history."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Score these two agent session logs against each other and tell me which performed better."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "score"

package/evals/evaluation-reports.md

@@ -0,0 +1,14 @@
+## Harness Evaluation Report - 2026-08-31T14:29:33.829Z
+
+| Dimension | Score |
+|---|---|
+| Correctness & Factuality | 10/10 |
+| Token & Step Efficiency | 5/10 |
+| Anti-loop & Focus | 10/10 |
+| Environment & Tool Awareness | 10/10 |
+| **Total Score** | **35/40** |
+
+### Insights
+Repository-grounded audit. Environment preflight passed. Core npm test and 9 mechanism suites passed after npm ci, but routing misclassified this macro evaluation request as Tier 1; waza was unavailable; behavioral validation has 3 invalid case files; existing 37 Harness-only live records are 26/37 pass and 9/15 pressure pass; the fresh Claude runner case was invalid because shell:true truncated the prompt to Add.; no causal benchmark case proved vanilla fail to Harness pass.
+
+---

package/evals/fable-discipline/eval.yaml

@@ -0,0 +1,26 @@
+name: fable-discipline-routing-eval
+description: Trigger precision eval for the fable-discipline skill.
+skill: fable-discipline
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did fable-discipline trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/fable-discipline/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-fable-discipline-001
+name: Off-scope prompt must not trigger fable-discipline
+description: A request outside fable-discipline's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Rename the variable x to y in config.js."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "discipline"
+        - "replan"

package/evals/fable-discipline/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-fable-discipline-001
+name: On-scope prompt triggers fable-discipline
+description: "Enforce fable-mode's context discipline as a shadow guard. Use when running staged multi-agent loops to prevent context bloat, cap per-stage output size, keep each stage within its physical token boundaries, and stop agents from reading files outside the current stage's scope."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Enforce stage discipline on this large parallel build-out with named workers and per-stage verification."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "stage"

package/evals/fable-mode/eval.yaml

@@ -0,0 +1,26 @@
+name: fable-mode-routing-eval
+description: Trigger precision eval for the fable-mode skill.
+skill: fable-mode
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did fable-mode trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/fable-mode/tasks/negative-tier2-no-model.yaml

@@ -0,0 +1,18 @@
+id: negative-fable-model-005
+name: Ordinary Tier 2 work does not select a fable model
+description: A normal Tier 2 bugfix must not route to a model-pinned fable entrypoint.
+tags:
+  - negative-trigger
+  - tier-2
+inputs:
+  prompt: "Fix this checkout bug with a regression test; keep the change to the affected module."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: no-model-pinned-route
+    config:
+      contains:
+        - "fable-opus"
+        - "fable-sonnet"
+        - "fable-haiku"

package/evals/fable-mode/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-fable-mode-001
+name: Off-scope prompt must not trigger fable-mode
+description: A request outside fable-mode's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Answer a quick question about CSS specificity."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "fable"
+        - "orchestrator"

package/evals/fable-mode/tasks/positive-haiku-mode.yaml

@@ -0,0 +1,16 @@
+id: positive-fable-haiku-002
+name: Explicit Haiku mode routes mechanical work
+description: "Stage large, multi-source or multi-session tasks through a written plan, named fable agents, failable per-stage checks, and skeptical delivery review; use fable-opus, fable-sonnet, or fable-haiku for explicit model selection."
+tags:
+  - positive-trigger
+  - fable-haiku
+inputs:
+  prompt: "Fable on haiku: convert the repository's JSON fixtures to the new stable format and verify every output."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: routes-haiku
+    config:
+      contains:
+        - "haiku"

package/evals/fable-mode/tasks/positive-opus-mode.yaml

@@ -0,0 +1,16 @@
+id: positive-fable-opus-004
+name: Explicit Opus mode routes macro orchestration
+description: "Stage large, multi-source or multi-session tasks through a written plan, named fable agents, failable per-stage checks, and skeptical delivery review; use fable-opus, fable-sonnet, or fable-haiku for explicit model selection."
+tags:
+  - positive-trigger
+  - fable-opus
+inputs:
+  prompt: "Fable on opus: orchestrate the cross-service migration, delegate artifacts to named workers, and cold-verify the final architecture decision."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: routes-opus
+    config:
+      contains:
+        - "opus"

package/evals/fable-mode/tasks/positive-sonnet-mode.yaml

@@ -0,0 +1,16 @@
+id: positive-fable-sonnet-003
+name: Explicit Sonnet mode routes reasoning work
+description: "Stage large, multi-source or multi-session tasks through a written plan, named fable agents, failable per-stage checks, and skeptical delivery review; use fable-opus, fable-sonnet, or fable-haiku for explicit model selection."
+tags:
+  - positive-trigger
+  - fable-sonnet
+inputs:
+  prompt: "Fable on sonnect: synthesize the API research, implement the bounded adapter, and report the named verification command."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: routes-sonnet
+    config:
+      contains:
+        - "sonnet"

package/evals/fable-mode/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-fable-mode-001
+name: On-scope prompt triggers fable-mode
+description: "Stage large, multi-source or multi-session tasks through a written plan, named fable agents, failable per-stage checks, and skeptical delivery review; use fable-opus, fable-sonnet, or fable-haiku for explicit model selection."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Execute this large multi-phase refactor of the persistence layer using parallel sub-agent workers."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "worker"

package/evals/find-skills/eval.yaml

@@ -0,0 +1,26 @@
+name: find-skills-routing-eval
+description: Trigger precision eval for the find-skills skill.
+skill: find-skills
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did find-skills trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/find-skills/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-find-skills-001
+name: Off-scope prompt must not trigger find-skills
+description: A request outside find-skills's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Run the test suite and report failures."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "skills.sh"
+        - "npx skills"

package/evals/find-skills/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-find-skills-001
+name: On-scope prompt triggers find-skills
+description: "Search the agent-skills ecosystem (skills.sh / npx skills) when no installed skill fits; verify quality, apply ephemerally via temp cache by default, and install permanently via npx skills add only with explicit user approval"
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Find a skill that can generate mermaid sequence diagrams from code."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "skill"

package/evals/git-commit/eval.yaml

@@ -0,0 +1,26 @@
+name: git-commit-routing-eval
+description: Trigger precision eval for the git-commit skill.
+skill: git-commit
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did git-commit trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/git-commit/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-git-commit-001
+name: Off-scope prompt must not trigger git-commit
+description: A request outside git-commit's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Explain the difference between git rebase and git merge."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "Conventional Commit"
+        - "staged"

package/evals/git-commit/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-git-commit-001
+name: On-scope prompt triggers git-commit
+description: "Generate Angular-style commit messages after verifying the environment, submodules, and staged files; use for explicit commit requests or concluded task phases."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Commit these staged changes with a conventional commit message following our house style."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "commit"

package/evals/grill-me/eval.yaml

@@ -0,0 +1,26 @@
+name: grill-me-routing-eval
+description: Trigger precision eval for the grill-me skill.
+skill: grill-me
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did grill-me trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/grill-me/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-grill-me-001
+name: Off-scope prompt must not trigger grill-me
+description: A request outside grill-me's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Summarize the contents of docs/philosophy.md."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "grill"
+        - "interview"

package/evals/grill-me/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-grill-me-001
+name: On-scope prompt triggers grill-me
+description: "Acts as a relentless challenger to stress-test plans and architectures by interrogating one question at a time, finding loopholes, combating AI sycophancy, updating the CONTEXT.md glossary in real time, and handing off resolved decisions to to-spec for formal specs/ADRs."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Grill me on my startup idea until the requirements are airtight before we build anything."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "question"

package/evals/grill-with-docs/eval.yaml

@@ -0,0 +1,26 @@
+name: grill-with-docs-routing-eval
+description: Trigger precision eval for the grill-with-docs skill.
+skill: grill-with-docs
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did grill-with-docs trigger exactly when it should?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/grill-with-docs/tasks/negative-trigger.yaml

@@ -0,0 +1,16 @@
+id: negative-grill-with-docs-001
+name: Off-scope prompt must not trigger grill-with-docs
+description: A request outside grill-with-docs's DO NOT USE FOR scope must leave it dormant.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Bump all dependency versions to latest."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "CONTEXT.md"
+        - "ADR"

package/evals/grill-with-docs/tasks/positive-trigger.yaml

@@ -0,0 +1,15 @@
+id: positive-grill-with-docs-001
+name: On-scope prompt triggers grill-with-docs
+description: "Challenge the plan against the existing domain model, sharpen terminology, and update CONTEXT.md and ADRs inline as decisions crystallise; use when stress-testing a plan against documented language and decisions."
+tags:
+  - positive-trigger
+inputs:
+  prompt: "Interview me about this domain and persist what we establish into CONTEXT.md and ADRs."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "ADR"

package/evals/harness-everything/eval.yaml

@@ -0,0 +1,26 @@
+name: harness-everything-router-eval
+description: Tier routing precision eval for the harness-everything entry router.
+skill: harness-everything
+version: "1.0"
+config:
+  trials_per_task: 1
+  timeout_seconds: 300
+  parallel: false
+  executor: copilot-sdk
+  model: claude-sonnet-4.6
+metrics:
+  - name: task_completion
+    weight: 0.7
+    threshold: 0.8
+    description: Did the router emit the correct tier (1/2/3) for each prompt class?
+  - name: efficiency
+    weight: 0.3
+    threshold: 0.7
+    description: Did the skill stay within behavior limits?
+graders:
+  - type: behavior
+    name: token-budget
+    config:
+      max_tokens: 1200
+tasks:
+  - "tasks/*.yaml"

package/evals/harness-everything/tasks/negative-non-software.yaml

@@ -0,0 +1,16 @@
+id: negative-non-software-001
+name: Non-software question bypasses routing
+description: Chat or general Q&A, and non-software writing, bypass the router entirely — no Checkpoint.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "What's the difference between latte and cappuccino?"
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "Routing Checkpoint"
+        - "Tier"

package/evals/harness-everything/tasks/negative-skill-indicated.yaml

@@ -0,0 +1,15 @@
+id: negative-skill-indicated-001
+name: Explicit skill indication bypasses routing
+description: A skill is already indicated, so the router must not re-route or print a Checkpoint.
+tags:
+  - negative-trigger
+inputs:
+  prompt: "Use the git-commit skill to commit my staged changes."
+expected:
+  should_trigger: false
+graders:
+  - type: text
+    name: not-contains-keywords
+    config:
+      contains:
+        - "Routing Checkpoint"

package/evals/harness-everything/tasks/positive-tier1.yaml

@@ -0,0 +1,16 @@
+id: positive-trivial-tier1-001
+name: Trivial edit routes to Tier 1
+description: "Route software/project work through the Harness kernel: classify Tier 1/2/3, establish mandatory invariants, then let the agent choose tactics and domain skills. Use for software triage, routing, and re-routing; not general Q&A or non-software writing."
+tags:
+  - positive-trigger
+  - tier-1
+inputs:
+  prompt: "Triage this new software task: fix the typo 'lorem ipsum' to 'lorem ipsum dolor' in docs/intro.md."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "Tier 1"

package/evals/harness-everything/tasks/positive-tier2.yaml

@@ -0,0 +1,16 @@
+id: positive-standard-tier2-001
+name: Standard bug fix routes to Tier 2
+description: "Route software/project work through the Harness kernel: classify Tier 1/2/3, establish mandatory invariants, then let the agent choose tactics and domain skills. Use for software triage, routing, and re-routing; not general Q&A or non-software writing."
+tags:
+  - positive-trigger
+  - tier-2
+inputs:
+  prompt: "Fix this bug: the checkout total double-counts coupon discounts when two coupons apply."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "Tier 2"

package/evals/harness-everything/tasks/positive-tier3.yaml

@@ -0,0 +1,16 @@
+id: positive-macro-tier3-001
+name: Architectural task routes to Tier 3
+description: "Route software/project work through the Harness kernel: classify Tier 1/2/3, establish mandatory invariants, then let the agent choose tactics and domain skills. Use for software triage, routing, and re-routing; not general Q&A or non-software writing."
+tags:
+  - positive-trigger
+  - tier-3
+inputs:
+  prompt: "Route this coding request: split our monolith into two services (orders and billing) with an event bus between them; this touches deployment, data migration, and every team's code."
+expected:
+  should_trigger: true
+graders:
+  - type: text
+    name: contains-keywords
+    config:
+      contains:
+        - "Tier 3"