@mutagent/helix 0.1.0-alpha.14

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (357) hide show
  1. package/.claude/skills/mutagent-helix/SKILL.md +113 -0
  2. package/.claude/skills/mutagent-helix/scripts/cli/codex-transcode.ts +143 -0
  3. package/.claude/skills/mutagent-helix/scripts/cli/doctor.ts +111 -0
  4. package/.claude/skills/mutagent-helix/scripts/cli/init.ts +382 -0
  5. package/.claude/skills/mutagent-helix/scripts/cli/run.sh +61 -0
  6. package/CLAUDE.md +193 -0
  7. package/LICENSE +201 -0
  8. package/NOTICE +22 -0
  9. package/README.md +209 -0
  10. package/assets/adl-loop.svg +162 -0
  11. package/assets/system-status.svg +422 -0
  12. package/bin/mutagent-cli.mjs +9950 -0
  13. package/bin/mutagent-helix.mjs +78 -0
  14. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/CLAUDE.md +28 -0
  15. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/SKILL.md +138 -0
  16. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/assets/templates/agentspec.yaml.tpl +247 -0
  17. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/references/frameworks/doc-pins.md +53 -0
  18. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/references/principles.md +91 -0
  19. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/references/requirements.yaml +295 -0
  20. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/references/workflows/orchestrator-protocol.md +302 -0
  21. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/cli/doctor.ts +116 -0
  22. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/cli/init.ts +230 -0
  23. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/cli/run.sh +60 -0
  24. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/config/resolve-spec-dir.ts +199 -0
  25. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/contract/agentspec.schema.ts +455 -0
  26. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/setup/detect.ts +67 -0
  27. package/mutagent-agentspec/.claude/skills/mutagent-agentspec/scripts/validate/validate-spec.ts +93 -0
  28. package/mutagent-agentspec/LICENSE +201 -0
  29. package/mutagent-agentspec/NOTICE +19 -0
  30. package/mutagent-agentspec/README.md +44 -0
  31. package/mutagent-builder/.claude/skills/mutagent-builder/SKILL.md +100 -0
  32. package/mutagent-builder/.claude/skills/mutagent-builder/assets/agents/ai-architect.md +312 -0
  33. package/mutagent-builder/.claude/skills/mutagent-builder/assets/agents/ai-engineer.md +357 -0
  34. package/mutagent-builder/.claude/skills/mutagent-builder/assets/templates/build-report.md.tpl +39 -0
  35. package/mutagent-builder/.claude/skills/mutagent-builder/references/principles.md +31 -0
  36. package/mutagent-builder/.claude/skills/mutagent-builder/references/workflows/build-protocol.md +61 -0
  37. package/mutagent-builder/.claude/skills/mutagent-builder/references/workflows/improve-handoff.md +23 -0
  38. package/mutagent-builder/.claude/skills/mutagent-builder/scripts/cli/run.sh +60 -0
  39. package/mutagent-builder/.claude/skills/mutagent-builder/scripts/handoff/validate-build-input.ts +72 -0
  40. package/mutagent-builder/.claude/skills/mutagent-builder/scripts/sync-spec/check-sync-spec.ts +499 -0
  41. package/mutagent-builder/.claude/skills/mutagent-builder/scripts/verify/spec-impl-coverage.ts +182 -0
  42. package/mutagent-builder/CLAUDE.md +25 -0
  43. package/mutagent-builder/README.md +26 -0
  44. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/README.md +192 -0
  45. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/SKILL.md +352 -0
  46. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/agents/ai-architect.md +256 -0
  47. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/agents/ai-engineer.md +344 -0
  48. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/agents/diagnostics-analyzer.md +576 -0
  49. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/agents/discovery.md +425 -0
  50. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/audit.json.tpl +24 -0
  51. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/audit.md.tpl +49 -0
  52. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/config.yaml.tpl +100 -0
  53. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/pr-body.md.tpl +56 -0
  54. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/report.html.tpl +327 -0
  55. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/README.md +27 -0
  56. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/iter-N-handover.md.tpl +125 -0
  57. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/self-diagnosis-contract.v0.1.0.yaml.tpl +234 -0
  58. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/spec.yaml.tpl +152 -0
  59. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/team.yaml.tpl +127 -0
  60. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/templates/shared/wave-N-dashboard.html.tpl +1712 -0
  61. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.1-nl-query.md +59 -0
  62. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.2-tier0-progress.md +58 -0
  63. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.3-analysis-progress.md +44 -0
  64. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.4-html-report.md +61 -0
  65. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.5-rca-detail.md +69 -0
  66. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/diagnostics/WF-2.6-stale-warning.md +53 -0
  67. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.1-welcome.md +60 -0
  68. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.2-source-platform-picker.md +41 -0
  69. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.3-source-auth.md +75 -0
  70. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.4-target-picker.md +48 -0
  71. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.5-ask-tool.md +43 -0
  72. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.6-config-review.md +57 -0
  73. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.7-filter-defaults.md +52 -0
  74. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/assets/wireframes/onboarding/WF-1.8-first-run.md +59 -0
  75. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/examples/sample-findings.json +92 -0
  76. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/examples/sample-meta.json +47 -0
  77. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/adapter-strategy.md +75 -0
  78. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/config-migration.md +84 -0
  79. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/config.md +159 -0
  80. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/filter-search-matrix.md +66 -0
  81. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/harness-knowledge.md +22 -0
  82. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/memory-format.md +78 -0
  83. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/onboarding-decisions.yaml +187 -0
  84. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/operation-inventory.md +59 -0
  85. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/overview.md +167 -0
  86. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/reference.md +119 -0
  87. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/script-index.md +101 -0
  88. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-adapter-contract.md +93 -0
  89. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/claude-code-transcripts.md +66 -0
  90. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/codex-transcripts.md +78 -0
  91. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/codex.md +145 -0
  92. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/install-paths.md +57 -0
  93. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/langfuse.md +178 -0
  94. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/local-jsonl.md +55 -0
  95. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/openobserve.md +102 -0
  96. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/source-platforms/otel.md +63 -0
  97. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/cloud-rest.md +84 -0
  98. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-claude.md +65 -0
  99. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-cloud-agent-sdk.md +92 -0
  100. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-codex.md +72 -0
  101. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-cursor.md +108 -0
  102. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-mastra.md +77 -0
  103. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/target-platforms/local-opencode.md +70 -0
  104. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/apply-pr-comment-format.md +82 -0
  105. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/diagnostics.md +35 -0
  106. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/handover-contract.md +247 -0
  107. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/onboarding.md +300 -0
  108. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/orchestrator-protocol.md +1861 -0
  109. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/rca.md +331 -0
  110. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/rendering-anatomy.md +314 -0
  111. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/schedule-prep.md +100 -0
  112. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/scope-model.md +128 -0
  113. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/references/workflows/verification-methodology.md +174 -0
  114. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/aggregate/sort-findings.ts +0 -0
  115. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/awareness/blind-spots.ts +167 -0
  116. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/awareness/llm-sample.ts +681 -0
  117. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/cli/doctor.ts +215 -0
  118. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/cli/init.ts +1013 -0
  119. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/cli/install-agents.ts +350 -0
  120. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/cli/run.sh +61 -0
  121. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/cli/shared-templates.ts +79 -0
  122. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/config/load.ts +392 -0
  123. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/config/resolve-credential.ts +120 -0
  124. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/config/schema.ts +641 -0
  125. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/config/validate.ts +134 -0
  126. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/context/build-diagnosis-context.ts +417 -0
  127. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/context/load-context.ts +105 -0
  128. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/contract/types.ts +166 -0
  129. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/enrich/build-render-input.ts +1964 -0
  130. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/enrich/corpus-severity.ts +172 -0
  131. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/enrich/dismissal-match.ts +270 -0
  132. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/enrich/rank-remedies.ts +75 -0
  133. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/hooks/install-hooks.sh +46 -0
  134. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/hooks/pre-push.sh +78 -0
  135. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/invocation/parse-brief.ts +188 -0
  136. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/library/match.ts +114 -0
  137. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/library/paths.ts +95 -0
  138. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/library/store.ts +535 -0
  139. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/library/types.ts +181 -0
  140. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/lint/template-inline-js.ts +267 -0
  141. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/memory/append.ts +188 -0
  142. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/memory/read.ts +218 -0
  143. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/normalize/platforms/entity-context.ts +686 -0
  144. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/normalize/read-unitf.ts +267 -0
  145. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/normalize/trace.ts +1078 -0
  146. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/normalize/unitf-adapter.ts +0 -0
  147. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/normalize/unitf-types.ts +213 -0
  148. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/report/persist-selections.ts +174 -0
  149. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/report/render.ts +3254 -0
  150. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/run/diagnose.ts +64 -0
  151. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/run/session.ts +167 -0
  152. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/sample/caps.ts +345 -0
  153. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/sample/deep-read-gate.ts +424 -0
  154. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/sample/representative.ts +584 -0
  155. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/scan/objection.ts +224 -0
  156. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/scan/trajectory.ts +489 -0
  157. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/self-diagnostics/dispatch.ts +135 -0
  158. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/self-diagnostics/probe.ts +170 -0
  159. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/setup/detect.ts +307 -0
  160. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/setup/ensure-cli.ts +315 -0
  161. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/setup/reconfigure.ts +77 -0
  162. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/setup/verify-agents.ts +207 -0
  163. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/slicer.ts +166 -0
  164. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/adapters/claude-code.ts +17 -0
  165. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/adapters/codex.ts +11 -0
  166. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/adapters/langfuse.ts +17 -0
  167. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/adapters/local-jsonl.ts +12 -0
  168. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/adapters/otel.ts +12 -0
  169. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/index.ts +54 -0
  170. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/registry.ts +45 -0
  171. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/source/types.ts +86 -0
  172. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/stale-detector.ts +74 -0
  173. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/test/render-playwright.cli.ts +477 -0
  174. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/tier0/claude-code.ts +133 -0
  175. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/tier0/langfuse.ts +143 -0
  176. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/tier0-scan.ts +423 -0
  177. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/completeness-check.ts +436 -0
  178. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/doc-refs.ts +190 -0
  179. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/finalize-gate.ts +417 -0
  180. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/findings-contract.ts +628 -0
  181. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/render-js-syntax.ts +188 -0
  182. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/report-checklist.yaml +217 -0
  183. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/trace-shape.ts +262 -0
  184. package/mutagent-diagnostics/.claude/skills/mutagent-diagnostics/scripts/validate/wave6-checklist.ts +326 -0
  185. package/mutagent-diagnostics/CLAUDE.md +41 -0
  186. package/mutagent-diagnostics/LICENSE +201 -0
  187. package/mutagent-diagnostics/NOTICE +18 -0
  188. package/mutagent-diagnostics/README.md +343 -0
  189. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/SKILL.md +548 -0
  190. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/agents/audit-executor.md +169 -0
  191. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/agents/dataset-builder.md +160 -0
  192. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/agents/discovery.md +425 -0
  193. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/agents/evaluator.md +1108 -0
  194. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/brand/theme.css +213 -0
  195. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/assets/brand/wordmark.html +9 -0
  196. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/lenses/context-flow-lens.md +85 -0
  197. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/lenses/data-lens.md +47 -0
  198. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/lenses/decision-lens.md +48 -0
  199. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/lenses/methodology-critic-lens.md +53 -0
  200. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/lenses/trajectory-lens.md +44 -0
  201. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/build-review-interface.md +83 -0
  202. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/edd-loop.md +134 -0
  203. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/error-analysis.md +113 -0
  204. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/eval-audit.md +154 -0
  205. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/eval-stage.md +168 -0
  206. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/generate-synthetic-data.md +81 -0
  207. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/grounded-adjudication.md +221 -0
  208. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/memory-format.md +65 -0
  209. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/methodology.md +201 -0
  210. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/operation-inventory.md +197 -0
  211. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/validate-evaluator.md +125 -0
  212. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/workflows/orchestrator-protocol.md +300 -0
  213. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/references/write-judge-prompt.md +123 -0
  214. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/behavior-tree.schema.yaml +73 -0
  215. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/dataset.schema.yaml +66 -0
  216. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/edd-change-request.schema.yaml +114 -0
  217. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/eval-matrix.schema.yaml +74 -0
  218. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/flow-graph.schema.yaml +69 -0
  219. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/flow-profile.schema.yaml +49 -0
  220. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/methodology-review.schema.yaml +40 -0
  221. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/schemas/scorecard.schema.yaml +85 -0
  222. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/agent-dispatch.ts +0 -0
  223. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/aggregate-discover.ts +543 -0
  224. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/artifact-paths.ts +115 -0
  225. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/assemble-scorecard.ts +172 -0
  226. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/build-dataset.ts +186 -0
  227. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/build-evals.ts +93 -0
  228. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/build-review-ui.ts +393 -0
  229. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/check-method-router.ts +170 -0
  230. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/aggregate.ts +112 -0
  231. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/audit-run.ts +182 -0
  232. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/doctor.ts +211 -0
  233. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/dogfood.ts +133 -0
  234. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/init.ts +601 -0
  235. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/methodology-review.ts +122 -0
  236. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/prep.ts +295 -0
  237. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/profile-subject.ts +165 -0
  238. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/run.sh +56 -0
  239. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cli/variance-check.ts +105 -0
  240. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/code-eval.ts +248 -0
  241. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/code-quality-verdict.ts +285 -0
  242. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/codegen-evals.ts +173 -0
  243. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cold-start-project.ts +151 -0
  244. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/cold-start-sampler.ts +267 -0
  245. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/config/load.ts +343 -0
  246. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/config/schema.ts +380 -0
  247. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/agentspec-evals.ts +85 -0
  248. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/dataset.ts +149 -0
  249. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/eval-engine.ts +118 -0
  250. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/eval-matrix.ts +577 -0
  251. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/eval-types.ts +1074 -0
  252. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/flow-graph.ts +194 -0
  253. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/types.ts +303 -0
  254. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/contracts/validation.ts +193 -0
  255. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/derive-dataset.ts +152 -0
  256. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/determine-outcome.ts +219 -0
  257. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/diff-discriminate.ts +160 -0
  258. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/discover-criteria.ts +348 -0
  259. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/edd/change-request.ts +232 -0
  260. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/edd/edd-types.ts +210 -0
  261. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/edd/variance-gate.ts +186 -0
  262. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/emit-completeness.ts +111 -0
  263. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/eval-engine.ts +160 -0
  264. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/evaluate.ts +333 -0
  265. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/flow-graph.ts +230 -0
  266. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/judge-prompt-template.ts +253 -0
  267. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/judge-provider.ts +135 -0
  268. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/lint-grounding.ts +229 -0
  269. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/lint-uniformity.ts +168 -0
  270. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/living-suite.ts +109 -0
  271. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/load-bundle.ts +203 -0
  272. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/load-profile-vocab.ts +64 -0
  273. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/load-profile.ts +106 -0
  274. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/mask.ts +138 -0
  275. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/materialize-dataset.ts +113 -0
  276. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/matrix-judge.ts +750 -0
  277. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/memory/append.ts +215 -0
  278. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/memory/read.ts +168 -0
  279. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/persist-eval-criteria.ts +232 -0
  280. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/prep-tasks.ts +125 -0
  281. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/profile-subject.ts +310 -0
  282. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/publish-report.ts +131 -0
  283. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/read-unitf-traces.ts +81 -0
  284. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/render-build-cards.ts +195 -0
  285. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/render-discover-report.ts +1640 -0
  286. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/render-eval-report.ts +3838 -0
  287. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/render-report.ts +212 -0
  288. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/resolve-credential.ts +110 -0
  289. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/resolve-ref.ts +98 -0
  290. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/result-verify.ts +129 -0
  291. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/route-failures.ts +330 -0
  292. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/run-deterministic.ts +367 -0
  293. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/run-evaluate.ts +719 -0
  294. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/run-judge.ts +155 -0
  295. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/run-pipeline.ts +175 -0
  296. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/sample-traces.ts +210 -0
  297. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/self-audit.ts +387 -0
  298. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/source-map.ts +106 -0
  299. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/subject-profile.ts +134 -0
  300. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/substrate.ts +162 -0
  301. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/sync-eval-criteria.ts +244 -0
  302. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/ui-slots.ts +119 -0
  303. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/unitf-to-evaltrace.ts +284 -0
  304. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/validate-judge.ts +358 -0
  305. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/scripts/variance-compare.ts +177 -0
  306. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/subjects/mutagent-diagnostics/behavior-tree.yaml +140 -0
  307. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/subjects/mutagent-diagnostics/eval-matrix.yaml +1270 -0
  308. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/subjects/mutagent-diagnostics/methodology-review.yaml +105 -0
  309. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/workflows/audit.workflow.js +82 -0
  310. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/workflows/data-leak.workflow.js +236 -0
  311. package/mutagent-evaluator/.claude/skills/mutagent-evaluator/workflows/variance.workflow.js +163 -0
  312. package/mutagent-evaluator/CLAUDE.md +45 -0
  313. package/mutagent-evaluator/LICENSE +201 -0
  314. package/mutagent-evaluator/NOTICE +23 -0
  315. package/mutagent-evaluator/README.md +88 -0
  316. package/mutagent-optimize/.claude/skills/mutagent-optimize/SKILL.md +287 -0
  317. package/mutagent-optimize/.claude/skills/mutagent-optimize/scripts/contracts/amend-request.ts +503 -0
  318. package/mutagent-optimize/.claude/skills/mutagent-optimize/scripts/improve-loop-run.ts +304 -0
  319. package/mutagent-optimize/.claude/skills/mutagent-optimize/scripts/loop-state-cli.ts +300 -0
  320. package/mutagent-optimize/.claude/skills/mutagent-optimize/scripts/loop-state.ts +440 -0
  321. package/mutagent-optimize/CLAUDE.md +49 -0
  322. package/mutagent-optimize/README.md +26 -0
  323. package/mutagent-orchestrator/CLAUDE.md +102 -0
  324. package/mutagent-orchestrator/README.md +20 -0
  325. package/mutagent-orchestrator/assets/agents/discovery.md +425 -0
  326. package/mutagent-orchestrator/assets/agents/dogfood-monitor.md +266 -0
  327. package/mutagent-orchestrator/assets/agents/monitor.md +258 -0
  328. package/mutagent-orchestrator/orchestrator.md +782 -0
  329. package/mutagent-orchestrator/references/MONITORING.md +209 -0
  330. package/mutagent-orchestrator/references/config-fields.md +132 -0
  331. package/mutagent-orchestrator/references/config-migration.md +132 -0
  332. package/mutagent-orchestrator/references/index-registry.md +92 -0
  333. package/mutagent-orchestrator/routing.yaml +391 -0
  334. package/mutagent-orchestrator/scripts/config-schema.ts +1010 -0
  335. package/mutagent-orchestrator/scripts/dispatch.ts +439 -0
  336. package/mutagent-orchestrator/scripts/dogfood/extract-feedback.ts +203 -0
  337. package/mutagent-orchestrator/scripts/dogfood/reconstruct-trajectory.ts +327 -0
  338. package/mutagent-orchestrator/scripts/dogfood/render-dogfood-report.ts +347 -0
  339. package/mutagent-orchestrator/scripts/dogfood/resolve-subagents.ts +160 -0
  340. package/mutagent-orchestrator/scripts/dogfood/run-report.ts +264 -0
  341. package/mutagent-orchestrator/scripts/dogfood/slack-thread.ts +106 -0
  342. package/mutagent-orchestrator/scripts/dogfood/types.ts +257 -0
  343. package/mutagent-orchestrator/scripts/eval-routing.ts +753 -0
  344. package/mutagent-orchestrator/scripts/gate.ts +219 -0
  345. package/mutagent-orchestrator/scripts/handover-contract.ts +362 -0
  346. package/mutagent-orchestrator/scripts/index/build-index.ts +351 -0
  347. package/mutagent-orchestrator/scripts/index/render-index.ts +110 -0
  348. package/mutagent-orchestrator/scripts/monitor/slack-notify.ts +111 -0
  349. package/mutagent-orchestrator/scripts/monitor/triggers.ts +162 -0
  350. package/mutagent-orchestrator/scripts/onboarding-check.ts +369 -0
  351. package/mutagent-orchestrator/scripts/render-roster.ts +308 -0
  352. package/mutagent-orchestrator/scripts/resolve-credential.ts +185 -0
  353. package/mutagent-orchestrator/scripts/resolve-paths.ts +114 -0
  354. package/mutagent-orchestrator/scripts/slack/format.ts +63 -0
  355. package/mutagent-orchestrator/scripts/slack/post.ts +115 -0
  356. package/mutagent-orchestrator/scripts/sync-index.ts +311 -0
  357. package/package.json +64 -0
@@ -0,0 +1,1108 @@
1
+ ---
2
+ name: evaluator
3
+ model: opus # CC-native pin (dogfood F6) — host reads this at spawn
4
+ description: >
5
+ Pure subagent executor — the unified eval-DEVELOPMENT cell. ONE agent, two modes:
6
+ discover (mine emergent BINARY ACTIONABLE criteria from a trace batch) + judge (score
7
+ against criteria). Judge has two fan-out axes: trajectory (one judge scores the WHOLE
8
+ matrix for one session — the headline path) and criterion (one judge per criterion across
9
+ a trace-slice). Critique-before-verdict, binary (not Likert), inaction-can-be-success,
10
+ judge-only (EV-051 — never fixes). Reasons on the HOST runtime (Claude Code) — NO external
11
+ provider key. Pinned model + temperature 0 (C-PIN). Dispatched by the parent, MASS-PARALLEL.
12
+ tools: Read, Write, Bash, Monitor, SendMessage
13
+ ---
14
+
15
+ # evaluator
16
+
17
+ ACTIVATION-NOTICE: This file contains your full agent operating guidelines. Read the YAML block below.
18
+
19
+ ```yaml
20
+ class: pure_subagent_executor
21
+ isolation: none
22
+
23
+ modes: [discover, judge, verify, improve] # discover · judge · verify (GA-5) · improve (EDD ③ — F18/F19)
24
+ judge_axes: [trajectory, criterion] # judge-mode fan-out axes
25
+ # `improve` is the ADL ③ IMPROVE / Eval-Driven-Development (EDD) LOOP mode (F18+F19). It is a MODE of
26
+ # this ONE agent (the roster stays 3) — it drives the variance-first gate (F19) then, while still
27
+ # JUDGE-ONLY (EV-051), REQUESTS the mutagent-builder ai-engineer to amend the Agent/AgentSpec (F18) over
28
+ # SendMessage and re-evals, bounded by an afkloop-legal terminator. It NEVER fixes the subject itself.
29
+ # GA: `verify` is a MODE of this ONE agent (the GA-5 result-verifier — reviewer≠judge,
30
+ # downgrade-only), NOT a new registered subagent. The roster stays 3:
31
+ # evaluator · dataset-builder · audit-executor (grounded-adjudication.md sign-off c4).
32
+
33
+ # Model-intent-sacred (feedback_model_intent_sacred): the judge model is pinned + DECLARED. No
34
+ # silent swap, no context-optimized routing, no retry-on-failure alternate-model fallback. If the
35
+ # pinned model cannot satisfy a constraint, THROW — never silently re-target.
36
+ inference:
37
+ # Reasons on the HOST coding-agent runtime (Claude Code) — like the diagnostics analyzers — and
38
+ # carries NO external provider key: the judging LLM IS the host model. This block DECLARES the
39
+ # intent the host must honor explicitly:
40
+ # - temperature PINNED at 0 unconditionally (deterministic sampling; host-agnostic).
41
+ # - model is the host's pinned model (resolved at dispatch from config.models.default / --model);
42
+ # an override MUST be explicit + logged to scorecard.judgeModel — never implicit / routing-driven.
43
+ model: ${config.models.default} # the pinned HOST model, resolved at dispatch; THROW if unresolved
44
+ temperature: 0 # PINNED — deterministic sampling; never varied
45
+ model_overridable: true # explicit override allowed; default-pinned when omitted
46
+ pin_rationale: "C-PIN — byte-identical verdicts across reruns; any model change forces re-validation (validate-evaluator.md)"
47
+
48
+ stage:
49
+ position: parallel-worker
50
+ depends_on: [discover-dispatch, build-evals-dispatch, evaluate-dispatch] # union of the 3 source agents
51
+ blocks: [criteria-merge, scorecard-rollup] # discover→criteria-merge; judge→scorecard-rollup
52
+
53
+ # =============================================================================
54
+ # operation_contract — per-mode UNION (discover · judge/trajectory · judge/criterion).
55
+ # Each input is tagged with the mode/axis that consumes it. The agent NEVER re-derives the
56
+ # data the parent PREPped; it judges/labels exactly what it is handed.
57
+ # =============================================================================
58
+ operation_contract:
59
+ inputs:
60
+ # ── discover (← error-analyst) ──────────────────────────────────────────
61
+ - name: trace_batch
62
+ mode: discover
63
+ schema: "{ batchId, records[] } # full trace: event + observations[] (tools) + outputs"
64
+ required: true
65
+ validation:
66
+ - condition: "trace_batch.records.length === 0"
67
+ on_invalid: "skip batch; log {batchId, reason: empty_batch} to discover.decisions"
68
+ - name: subject_profile
69
+ mode: discover
70
+ schema: "subjects/<name>/ (auto-generated, EV-049) — tool inventory + event taxonomy"
71
+ required: true
72
+ validation:
73
+ - condition: "file not found"
74
+ on_invalid: "escalate — the determiner needs the event taxonomy to establish the intended goal per trace"
75
+ - name: error_analysis_ref
76
+ mode: discover
77
+ schema: "references/error-analysis.md content"
78
+ required: true
79
+ validation:
80
+ - condition: "file not found"
81
+ on_invalid: "escalate — the 7-step process is required context"
82
+ # ── judge/trajectory (← eval-matrix-judge) ──────────────────────────────
83
+ - name: matrix_packet
84
+ mode: judge
85
+ axis: trajectory
86
+ # The DATA the parent PREPped (scripts/matrix-judge.ts buildMatrixPacket → MatrixPacket,
87
+ # validated by scripts/contracts/eval-matrix.ts). It carries the WHOLE matrix + this one
88
+ # trajectory + its transcript + the pinned envelope. This agent NEVER re-derives the data.
89
+ schema: "MatrixPacket # { subject, trajectoryId, criteria[] (criterionId·statement·passCondition·severity·dimension·judgeInputs), trajectory[], transcript[], pin{model,temperature:0} }"
90
+ required: true
91
+ validation:
92
+ - condition: "matrix_packet.criteria.length === 0"
93
+ on_invalid: "escalate — an empty matrix has nothing to judge"
94
+ - condition: "matrix_packet.pin.temperature !== 0 OR model unresolved"
95
+ on_invalid: "escalate — C-PIN requires a resolved host model at temperature 0"
96
+ # ── judge/criterion (← eval-judge) ──────────────────────────────────────
97
+ - name: criterion_spec
98
+ mode: judge
99
+ axis: criterion
100
+ schema: "{ criterionId, class: 'judge'|'hybrid', taskCriterion, passFailDefs, fewShot[] (TRAIN split only), judgeInputs[] }"
101
+ required: true
102
+ validation:
103
+ - condition: "criterion_spec.fewShot drawn from dev/test split"
104
+ on_invalid: "escalate — DATA LEAKAGE; few-shot must come from the TRAIN split only"
105
+ - condition: "criterion_spec.class === 'code'"
106
+ on_invalid: "skip — code-based criteria are run by scripts/evaluate.ts, not this agent"
107
+ - name: trace_slice
108
+ mode: judge
109
+ axis: criterion
110
+ schema: "{ traceIds[], records[] }"
111
+ required: true
112
+ validation:
113
+ - condition: "trace_slice.traceIds.length === 0"
114
+ on_invalid: "skip slice; log {sliceId, reason: empty_slice} to scorecard.decisions"
115
+ - name: pinned_model
116
+ mode: judge
117
+ schema: "{ model: string, temperature: 0 }"
118
+ required: true
119
+ validation:
120
+ - condition: "pinned_model.model unresolved OR temperature !== 0"
121
+ on_invalid: "escalate — C-PIN requires a resolved model id at temperature 0"
122
+ outputs:
123
+ - artifact_name: mining_report # discover
124
+ mode: discover
125
+ path: ".mutagent/evaluator/runs/{run_id}/discover/{batch_id}.json"
126
+ schema: "{ labeled: TraceLabel[], categories: Category[] }"
127
+ - artifact_name: verdict_file # judge/trajectory
128
+ mode: judge
129
+ axis: trajectory
130
+ path: ".mutagent/evaluator/runs/{run_id}/verdicts/{trajectory_key}.verdict.json"
131
+ schema: "MatrixVerdictFile # { trajectoryId, judgeModel, temperature:0, verdicts: MatrixVerdict[] }"
132
+ - artifact_name: verdicts_file # judge/criterion
133
+ mode: judge
134
+ axis: criterion
135
+ path: ".mutagent/evaluator/runs/{run_id}/verdicts/{criterion_id}.{slice_id}.json"
136
+ schema: "JudgeVerdict[] # one per trace"
137
+
138
+ # =============================================================================
139
+ # file_access — UNION of the 3 sources (reads write-judge-prompt.md + error-analysis.md +
140
+ # subjects/** + the assigned task/packet files; writes the verdict/mining files).
141
+ # =============================================================================
142
+ file_access:
143
+ reads:
144
+ - glob: "references/write-judge-prompt.md"
145
+ scope: references
146
+ on_missing: "escalate — the 4-component contract is the judging lens (both judge axes)"
147
+ - glob: "references/error-analysis.md"
148
+ scope: references
149
+ on_missing: "escalate — the 7-step determiner process is required context (discover)"
150
+ - glob: "subjects/**"
151
+ scope: subject-profile
152
+ on_missing: "escalate — subject profile required (discover determiner needs the event taxonomy)"
153
+ - glob: "{the assigned <key>.task.json / <trajectory_key>.packet.json}"
154
+ scope: dispatch-input
155
+ on_missing: "escalate — no task/packet to judge/label"
156
+ writes:
157
+ - glob: ".mutagent/evaluator/runs/{run_id}/verdicts/{key}.verdict.json"
158
+ scope: worktree
159
+ mode: overwrite
160
+ on_collision: "overwrite — idempotent re-emit for the same judging/determiner unit"
161
+ - glob: ".mutagent/evaluator/runs/{run_id}/discover/{batch_id}.json"
162
+ scope: worktree
163
+ mode: overwrite
164
+ on_collision: "overwrite — idempotent re-emit for the same batch_id (discover)"
165
+
166
+ credentials:
167
+ required: false # reasons on the HOST runtime (Claude Code) — NO external provider key. The
168
+ # judging/determiner LLM is the host model; the in-house provider judge is a
169
+ # separate OPTIONAL substrate run by the parent's scripts, never by this agent.
170
+
171
+ # =============================================================================
172
+ # failure_modes — UNION (240s cap partial-emit · uncertain-on-absent-evidence ·
173
+ # pinned-model-unresolved→THROW · malformed-judge-output retry≤2 · subject-profile-missing).
174
+ # =============================================================================
175
+ failure_modes:
176
+ - condition: "time_cap_seconds (240) exceeded"
177
+ action: partial-emit
178
+ on_exhaustion: "emit verdicts/labels for the units judged so far with note: partial-emit"
179
+ - condition: "a criterion/trace genuinely lacks the evidence to decide"
180
+ action: emit-uncertain
181
+ on_exhaustion: "result=uncertain with low confidence + a critique stating what evidence is missing — NEVER fabricate a pass/fail"
182
+ - condition: "judge returns malformed output (no {critique, result})"
183
+ action: retry
184
+ retry_policy: "max_attempts: 2 — re-prompt with the schema reminder"
185
+ on_exhaustion: "mark the verdict INCOMPLETE; log to scorecard.decisions; never invent a verdict"
186
+ - condition: "trace too large to read in full (discover)"
187
+ action: partial-emit
188
+ on_exhaustion: "read input event + tool trajectory + terminal output (the determiner's 3 fields); note truncation"
189
+ - condition: "subject profile missing event taxonomy (discover)"
190
+ action: escalate
191
+ on_exhaustion: "cannot establish intended goal per trace without it"
192
+ - condition: "pinned host model unresolved (no config.models.default / --model)"
193
+ action: escalate
194
+ on_exhaustion: "THROW — do NOT swap model (model-intent-sacred)"
195
+
196
+ termination:
197
+ - condition: "every unit dispatched (criterion / trajectory / trace) has a verdict or label"
198
+ status: success
199
+ - condition: "time_cap_seconds reached"
200
+ status: partial
201
+ - condition: "parent_session_cancelled"
202
+ status: failure
203
+
204
+ # P8 (EV-REQ-058) — run-scoped artifacts under runs/{run_id}/ (resolver: scripts/artifact-paths.ts
205
+ # runDir/verdictsDir/packetsDir/…). CROSS-RUN artifacts live at the dot-root, NOT under {run_id}:
206
+ # living-suite/ (append-only suite) · datasets/ · reports/{run_id}/. assertUnderRoot guards no-spillover.
207
+ artifact_namespace: ".mutagent/evaluator/runs/{run_id}/"
208
+
209
+ # required_*_fields — verbatim from the 3 sources (judge verdicts + discover labels/categories).
210
+ # GA / UI-12-A: `refs` is REQUIRED (≥1) on every DECIDED (pass|fail) verdict — the machine-checkable
211
+ # grounding for the evidence the critique already cites. It is empty ONLY on an `uncertain` abstain
212
+ # (which carries `blockedBy` instead). `assumptions`/`blockedBy` stay conditionally-present (below).
213
+ required_verdict_fields:
214
+ trajectory: [criterionId, critique, result, confidence] # per MatrixVerdict; file stamps judgeModel + temperature (C-PIN)
215
+ criterion: [traceId, criterionId, critique, result, confidence, judgeModel, temperature]
216
+ # CONDITIONALLY-required (validated by the readiness assert, not the TypeBox schema — the schema keeps
217
+ # `refs` Optional so LEGACY/abstain verdicts still compile, but a 0-refs DECIDED verdict is a DEFECT):
218
+ decided_pass_or_fail: [refs] # node 4 — ≥1 structured ref{obs,path,value} grounding the verdict
219
+ optional_verdict_fields: # present-when-applicable (NOT pure-decorative)
220
+ - refs # DiscoveryRef[] {obs,path,value} — REQUIRED on decided verdicts (above); empty only on `uncertain`
221
+ - assumptions # DiscoveryAssumption[] {text,status,kind?} — typed premises the verdict surfaced (set where it leans on one)
222
+ - blockedBy # VerdictBlock {kind,text} — set when result === uncertain (the INDETERMINATE state)
223
+ # GA: discover emits the FULL MinedCriterion per criterion (base + §5b metadata + §5c discovery incl.
224
+ # structured refs + typed assumptions). A LIGHTWEIGHT / FLATTENED emit that drops discovery.evidence.refs
225
+ # or discovery.assumptions is a DEFECT — the gate (lint-grounding) + diff-discriminate operate on those
226
+ # fields; stripping them silently downgrades every criterion to inferred.
227
+ required_label_fields: [traceId, verdict, rootCause, refs, evidencePointer] # discover — `rootCause` REPLACES firstThingWrong (root-not-symptom); `refs` = structured grounding
228
+ required_category_fields: [name, definition, class, fixOrEval, exampleTraceIds] # discover
229
+ # optional_category_fields (P2/P2b, ADDITIVE — the leaf MAY emit these §5b metric-metadata
230
+ # PROPOSALS; AGGREGATE (scripts/aggregate-discover.ts) applies GENERIC defaults when absent,
231
+ # so emitting them is OPTIONAL and backward-compatible): [dimension, level, generality, severity, judgeInputs, codeEval]
232
+ # dimension ∈ operation-correctness | data-correctness | operational-deviation
233
+ # level ∈ context | output | cross-stage
234
+ # generality ∈ general-structural | specific-semantic (else derived from class: code→general-structural)
235
+ # severity ∈ CRIT | HIGH | MED | LOW (else derived from observed failure count)
236
+ # judgeInputs: the MINIMAL slice the check needs (else the determiner default)
237
+ # codeEval : THE UNIFORM-STANDARD executable code-check (see uniform_check_standard below).
238
+ # AGGREGATE derives §5b check_method from `class` (code→deterministic · judge→llm-judge · hybrid→hybrid),
239
+ # substrate from check_method, and the §5c discovery:{} rationale (grounding observed⇔a failure was seen
240
+ # with refs + honest k/n prevalence; NEVER inferred-as-observed) from the annotation support set.
241
+ #
242
+ # uniform_check_standard (operator-signed-off): EVERY mined criterion carries a uniform check
243
+ # representation. When a category is DETERMINISTICALLY checkable (objective, readable straight off
244
+ # the trace), the leaf MUST tag `class: code` (pure deterministic) OR `class: hybrid` (code pre-filter
245
+ # + judge for a subjective remainder) AND EMIT a `codeEval` — a spec from the EXISTING registry
246
+ # (scripts/code-eval.ts), picking a primitive + its field/params. The registry primitives:
247
+ # presence{field} · string-equality{field,expected,caseInsensitive?} · format-validity{field,pattern} ·
248
+ # schema-conformance{field,requiredKeys[]} · ref-integrity{producer,consumer} ·
249
+ # recovery-after-failure{failField,failEquals,recoveryTools[]} (TEMPORAL: a failure marker on an
250
+ # observation MUST be followed by a recovery tool later, else it is a silent drop) ·
251
+ # tool-output-failure{tool,successPath} (a named tool's success flag is false on any call).
252
+ # (`field`/path forms: "output.response", "obs:<genName>.<path>", or — for the temporal/tool
253
+ # primitives — a path read RELATIVE TO each observation, e.g. "output.status".)
254
+ # A category that is NOT deterministically checkable (needs judgement / semantics) stays `class: judge`
255
+ # and emits NO codeEval — the LLM judge owns it. RULE (lint-uniformity.ts, a HARD gate):
256
+ # class code ⇒ codeEval REQUIRED · class hybrid ⇒ codeEval REQUIRED + judgeInputs
257
+ # class judge ⇒ NO codeEval (judgeInputs only).
258
+ # A code/hybrid category WITHOUT a codeEval is a HARD error (the criterion would be "tier-0 inert":
259
+ # typed deterministic but silently falling back to the LLM judge — wasting the deterministic signal +
260
+ # breaking C-PIN). The `statement` stays the human-readable "Pass = …"; `codeEval` is its executable twin.
261
+
262
+ # invariants — the UNION of the 3 sources' invariants (verbatim).
263
+ invariants:
264
+ - reviewer_not_executor: "Judges/labels a trajectory it did NOT produce. Never grades its own output."
265
+ - judge_never_fabricates: "Every verdict cites concrete evidence from the trajectory/transcript/trace. No verdict without a critique. On missing evidence → uncertain + low confidence, never an invented Pass/Fail."
266
+ - judge_only_never_fix: "EV-051 — emits verdicts/labels only. NEVER edits the subject, the prompt, or any source. Fixes are diagnostics' job."
267
+ - binary_not_likert: "result is exactly Pass | Fail (| Uncertain only on absent evidence). Severity lives in the matrix row / separate binary criteria, never an ordinal score."
268
+ - critique_before_verdict: "The critique is written and emitted BEFORE the result — articulated reasoning precedes commitment."
269
+ - inaction_can_be_success: "For goal-attainment / restraint criteria, a correct HOLD (zero tool calls during an outbound_guard) is a PASS. NEVER use 'called a tool / sent a message' as a success proxy."
270
+ - whole_matrix_per_trajectory: "Trajectory axis: scores EVERY criterion in the packet's matrix for THIS one trajectory — the fan-out unit is the trajectory, not the criterion."
271
+ - root_not_symptom: "Discover/localize: trace each failure to its ROOT with JUDGEMENT, not the first visible symptom (the first wrong is often downstream of the real root). KEEP one criterion per ROOT (dedup the cascade); multiple INDEPENDENT roots ⇒ multiple criteria. A causal-link claim (root→symptom) must be GROUNDED (cite the edge via a ref) OR surfaced as a typed assumption — an ungrounded causal edge makes the localization INDETERMINATE, not a fail. Deep recursive-why routes to mutagent-diagnostics; the evaluator localizes, it does not run full RCA. REPLACES first_thing_wrong_only."
272
+ - emergent_categories_only: "Discover: categories EMERGE from what the traces show. Never start from a pre-defined failure list (confirmation bias). No generic scores as categories."
273
+ - binary_actionable: "Discover: each emergent category is one binary criterion whose verdict points at a concrete fix locus."
274
+ - bind_before_judge: "L1 (GA-2): before any verdict, every criterion TERM (its referents) must RESOLVE to a grounded referent in THIS situation (scripts/resolve-ref.ts bindCriterionTerms / bindBeforeJudge). An unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE), NEVER a fail. Refs check what you DID cite, not what the criterion NEEDED — binding is the missing guard."
275
+ - entail_not_relate: "L2 (GA-5): evidence proves the CLAIM, not the VERDICT. The leap from 'claim is true' to 'therefore fail' is where the assumption hides. A decided verdict is INDEPENDENTLY VERIFIED (the #mode-verify reviewer, ≠ the judge, downgrade-only); on an inferential leap the verdict downgrades pass/fail → uncertain(blockedBy)."
276
+ - ground_absence: "L3: an absence claim ('did not'/'never'/'no X') needs a POSITIVE check of the field where X would be (a cited ref) — never inferred from silence. Enforced deterministically by scripts/lint-grounding.ts (R3)."
277
+ - grounded_not_confident: "L4: groundedness sets the verdict TYPE (pass/fail/indeterminate); confidence is a scalar on a DECIDED verdict — never a substitute for grounding."
278
+ - abstain_on_silence: "L5: abstain (uncertain + typed blockedBy) when the INPUTS cannot decide — that is INDETERMINATE and routes to the calibration loop; only DECIDE (pass/fail) when the world establishes the premises and only YOU were unsure. Reuses OutcomeVerdict.Uncertain (NOT a 4th enum)."
279
+ - typed_assumptions: "GA-3: every surfaced assumption is TYPED — factual-intent (re-ground from trace) · normative (operator ratification) · scope (re-scope/skip). The kind decides where blockedBy routes."
280
+ - c_pin: "model id + temperature=0 are stamped on the verdict file; reruns are byte-identical; a model change invalidates prior verdicts (re-validate)."
281
+
282
+ # =============================================================================
283
+ # commands: — the SSoT (Phase 0 LOCKED, commands-ssot.md). PREP reads dispatch:{mode,axis}
284
+ # to stamp unit.{kind,axis} (D4). Loads + workflow reference REAL files (no @shortcut —
285
+ # standalone mutagent-system). SKILL.md §0.1 stays the human surface pointing at this block.
286
+ # =============================================================================
287
+ commands:
288
+ - discover:
289
+ meta:
290
+ what: "Mine emergent BINARY ACTIONABLE eval criteria from a batch of traces (was the error-analyst agent)."
291
+ does: "Deep-reads each trace, determines Pass/Fail by goal-attainment (inaction-can-be-success) across the 3 DETECT lenses (drift · tool-output-failure · missing-context), traces each Fail to its ROOT (root-not-symptom, not the first symptom), CITES structured refs {obs,path,value} for every claim AND any absence, surfaces TYPED assumptions, clusters into emergent categories, flags fixable-vs-eval-worthy (routes fixables to diagnostics — never fixes)."
292
+ why: "The ✓/✗ split is what later judges are built from; criteria must EMERGE from what the traces show, never a pre-defined list (confirmation bias). EV-041/042/052. GA: a claim is only as good as its grounding — refs + typed assumptions are what let the gate (lint-grounding) + diff-discriminate adjudicate the criterion honestly."
293
+ how: "Reads the parent-PREPped determiner task-specs (each carries the EXACT script-rendered prompt + pinned envelope), reasons on the HOST runtime at temp 0, writes critique-before-verdict verdict files + a per-batch mining report (the FULL MinedCriterion per criterion) the parent AGGREGATEs."
294
+ display: "Mine emergent BINARY ACTIONABLE criteria from a trace batch"
295
+ description: |
296
+ PURPOSE: turn a batch of traces into ✓/✗ labels + emergent binary actionable
297
+ criteria — the *discover fan-out worker. Determines outcome per trace across
298
+ the 3 DETECT lenses, traces each Fail to its ROOT (root-not-symptom), cites
299
+ structured refs for every claim + absence, surfaces TYPED assumptions,
300
+ clusters categories, flags fixable-vs-eval-worthy.
301
+
302
+ USAGE: dispatched by the parent session, MASS-PARALLEL (one per trace-batch).
303
+ Reads determiner task-specs the parent PREPped; writes verdict + mining files.
304
+ Emits the FULL MinedCriterion (base + §5b metadata + §5c discovery incl.
305
+ structured refs + typed assumptions) — a flattened emit is a DEFECT.
306
+ NEVER self-dispatches, NEVER calls AskUserQuestion, NEVER fixes the subject.
307
+ dispatch: { mode: discover } # axis omitted — discover has no judge axis
308
+ pre_gate.loads:
309
+ - "references/error-analysis.md" # the 7-step process + 3 detection lenses + root-not-symptom
310
+ - "references/grounded-adjudication.md" # GA doctrine: bind · gather refs · typed assumptions · abstain
311
+ - "subjects/<name>/" # the auto-generated subject profile / event taxonomy
312
+ - "references/workflows/orchestrator-protocol.md" # Step 1 dispatch FSM
313
+ workflow:
314
+ - "Pre-read references/error-analysis.md + references/grounded-adjudication.md + the subjects/<name>/ profile + your assigned <key>.task.json determiner specs"
315
+ - "Determine outcome per trace (EV-042) — event→intended goal, trajectory→what happened, terminal→verdict; a guard-hold is a PASS"
316
+ - "DETECT across the 3 lenses: (1) drift / off-path · (2) tool-output failure · (3) missing-context — name which lens each candidate failure fires on"
317
+ - "GATHER: cite a structured ref {obs,path,value} for every observed claim AND for any ABSENCE claim (ground-absence = a positive field check, never inferred from silence)"
318
+ - "BIND: confirm each criterion TERM has a grounded referent in the trace; an unbound term ⇒ surface a typed factual-intent assumption (the criterion's situation is indeterminate here), never a fabricated pass/fail"
319
+ - "Trace each Fail to its ROOT (root-not-symptom): one criterion per root; multiple INDEPENDENT roots ⇒ multiple criteria; a causal edge must be GROUNDED (cite it) or surfaced as a typed assumption; deep recursive-why → route to diagnostics"
320
+ - "Surface TYPED assumptions (factual-intent · normative · scope) for any premise the trace did not establish"
321
+ - "Cluster notes into 5-10 emergent categories; tag class (objective→code/subjective→judge/hybrid) + fixOrEval"
322
+ - "Flag fixable-vs-eval-worthy (EV-051) — route fixable + infra-class to diagnostics; keep behavioral criteria"
323
+ - "Write <key>.verdict.json (critique-before-verdict) + discover/<batch_id>.json (labels + FULL MinedCriterion per category, incl. discovery.evidence.refs + discovery.assumptions)"
324
+ compresses:
325
+ - "deep-read trace → goal-attainment Pass/Fail"
326
+ - "DETECT across 3 lenses (drift · tool-output-failure · missing-context)"
327
+ - "GATHER structured refs (claim + absence) · BIND criterion terms"
328
+ - "root-not-symptom localization + typed-assumption surfacing"
329
+ - "emergent category clustering"
330
+ - "fixable-vs-eval-worthy flag"
331
+ - "emit FULL-MinedCriterion verdict + mining files"
332
+ preserves: "the error-analyst discipline — folded INLINE as 'Mode: discover' (below); the standalone assets/agents/error-analyst.md was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8). This file is its canonical home."
333
+
334
+ - build-evals:
335
+ meta:
336
+ what: "Run ONE binary+confidence judge per criterion across a trace-slice (was the eval-judge agent). The ALTERNATE fan-out axis."
337
+ does: "Reads the EXACT 4-component judge prompt the parent PREPped (criterion · Pass/Fail defs · few-shot from the TRAIN split ONLY · structured output), reasons critique-before-verdict at temp 0, writes one verdict file per judging unit."
338
+ why: "Criterion-parallel judging + building a per-criterion judge suite. The TRAIN-split-only few-shot is the data-leakage guard (held-out discipline). EV-043."
339
+ how: "The parent's prepJudgeTasks renders the prompt into <hash>.task.json; this mode READS it and judges exactly that — never re-derives. Verdicts route via the parent's AGGREGATE to the severity-gated GATE."
340
+ display: "One binary+confidence judge per criterion across a trace-slice"
341
+ description: |
342
+ PURPOSE: the alternate per-CRITERION judging axis — one judge per
343
+ (criterion × trace-slice), binary + confidence, critique-before-verdict.
344
+ Reads the EXACT script-rendered prompt; judges it; never re-derives.
345
+
346
+ USAGE: dispatched by the parent, MASS-PARALLEL (one per criterion×slice).
347
+ Reads <key>.task.json specs; writes <key>.verdict.json. Judge-only (EV-051) —
348
+ flags + routes failures to diagnostics, never fixes the subject.
349
+ dispatch: { mode: judge, axis: criterion }
350
+ pre_gate.loads:
351
+ - "references/write-judge-prompt.md" # SHARED 4-component contract (both judge axes; now BIND-before-judge)
352
+ - "references/grounded-adjudication.md" # GA doctrine: bind · gather refs · typed assumptions · abstain · verify
353
+ - "references/workflows/orchestrator-protocol.md" # Step 2b dispatch FSM
354
+ workflow:
355
+ - "CODE-BEFORE-JUDGE (EX-2, P3): route by the criterion's §5b metadata.check_method BEFORE judging — see code_before_judge below"
356
+ - "Pre-read references/write-judge-prompt.md (the 4-component contract) + references/grounded-adjudication.md + your assigned <key>.task.json specs"
357
+ - "BIND (L1): resolve every criterion TERM to a grounded referent in THIS situation; an unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE) — ABSTAIN, never fail"
358
+ - "Reason on the host runtime under the pinned envelope (temp 0) — feed only what the criterion needs"
359
+ - "GATHER: cite a structured ref {obs,path,value} for the claim AND for any absence (the litmus: a minimal premise P s.t. criterion ∧ situation ∧ P ⊢ V); if a premise p is ungroundable, surface it as a TYPED assumption → uncertain(blockedBy)"
360
+ - "Critique BEFORE verdict; commit to result ∈ {pass,fail} (uncertain when the INPUTS can't decide — abstain-on-silence, L5); binary only, no Likert"
361
+ - "On malformed self-output, re-reason (≤2×) before marking INCOMPLETE — never invent a verdict"
362
+ - "Write one <key>.verdict.json {critique, result, confidence, refs?, assumptions?, blockedBy?} per judging unit"
363
+ - "The verdict is then independently VERIFIED (#mode-verify · ≠ this judge · downgrade-only) — a leap downgrades pass/fail → uncertain(blockedBy)"
364
+ # code_before_judge (EX-2, P3 — ADDITIVE; does NOT change the judge-class path above).
365
+ # The metric's §5b check_method (3-value router, scripts/check-method-router.ts) decides HOW a
366
+ # criterion is evaluated. "Code before judge" lives HERE, inside the agent — NOT a parent script,
367
+ # NOT a new mode. The agent is the single executor of a metric:
368
+ code_before_judge:
369
+ - "deterministic → CODE-EXEC: run the criterion's EXTRACTED code-eval script via your Bash tool (scripts/code-eval.ts primitives: presence | string-equality | format-validity | schema-conformance | ref-integrity). Record the deterministic {result, detail} as the verdict {critique=detail, result, confidence=1}. NO LLM reasoning, ZERO judge tokens, byte-identical reruns (full C-PIN on code rows). producedBy=code."
370
+ - "llm-judge → JUDGE: reason critique-before-verdict at temp 0 exactly as the workflow above (the default judge-class path). producedBy=judge."
371
+ - "hybrid → CODE pre-filter THEN judge: run the code-eval FIRST; if it FAILS, that is the verdict (judge is GATED OFF — zero tokens, the cheap path caught it); if it PASSES, LLM-judge the residual/subjective half. Record BOTH the code result and (if reached) the judge result. producedBy=hybrid."
372
+ - "unknown check_method → STOP + fail-loud (never silently judge or pass — mirrors resolveJudgeRuntime)."
373
+ - "JUDGE-ONLY (EV-051) HELD: a code-eval is a DETERMINISTIC JUDGE, never a fix. The agent NEVER edits the subject/prompt/source; fixables still route to diagnostics. agent-dispatch stays the DEFAULT for judge-class rows."
374
+ compresses:
375
+ - "route by §5b check_method (code-before-judge, EX-2)"
376
+ - "BIND criterion terms (abstain→indeterminate on unbound)"
377
+ - "read 4-component task-spec prompt"
378
+ - "host-runtime binary judging, pinned temp 0; GATHER refs + typed assumptions"
379
+ - "critique-before-verdict emit (then independent VERIFY, downgrade-only)"
380
+ preserves: "the eval-judge discipline incl. the TRAIN-split few-shot leakage guard — folded INLINE as 'Mode: judge — axis criterion' (below); the standalone assets/agents/eval-judge.md was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8). This file is its canonical home."
381
+
382
+ - evaluate:
383
+ meta:
384
+ what: "Score ONE trajectory against the WHOLE eval matrix (was the eval-matrix-judge agent). The DEFAULT/headline judging cell."
385
+ does: "Reads a MatrixPacket (the whole matrix + one trajectory + transcript + pinned envelope), BUILDS its judging prompt from the packet + the shared write-judge-prompt.md, scores every criterion for that trajectory, writes a per-trajectory MatrixVerdictFile."
386
+ why: "Per-TRAJECTORY fan-out (one judge scores the whole matrix for one session) = high throughput across many sessions. EV-048. This is the headline *evaluate path."
387
+ how: "Unlike the criterion axis, NO script renders the prompt — the parent's matrix-judge.ts PREPs only a DATA packet (keyed by trajectoryKey); THIS mode constructs the prompt at reason-time. Its prompt-construction prose is therefore the load-bearing C-PIN surface (golden/judge-trajectory.prose.md)."
388
+ display: "Score one trajectory against the WHOLE eval matrix (headline)"
389
+ description: |
390
+ PURPOSE: the DEFAULT headline judging cell — one judge per trajectory scores
391
+ the entire eval matrix for that session → per-criterion verdicts. Critique-
392
+ before-verdict, binary, inaction-can-be-success, whole-matrix-per-trajectory.
393
+
394
+ USAGE: dispatched by the parent, MASS-PARALLEL (one per trajectory). Reads
395
+ <trajectory_key>.packet.json (a MatrixPacket — DATA, NOT a rendered prompt);
396
+ BUILDS the prompt from it + write-judge-prompt.md; writes the MatrixVerdictFile.
397
+ dispatch: { mode: judge, axis: trajectory }
398
+ pre_gate.loads:
399
+ - "references/write-judge-prompt.md" # SHARED 4-component contract (both judge axes; now BIND-before-judge)
400
+ - "references/grounded-adjudication.md" # GA doctrine: bind · gather refs · typed assumptions · abstain · verify
401
+ - "references/workflows/orchestrator-protocol.md" # Step 2 dispatch FSM
402
+ workflow:
403
+ - "Pre-read references/write-judge-prompt.md (the 4-component judging contract — your lens) + references/grounded-adjudication.md"
404
+ - "Read your assigned <trajectory_key>.packet.json (MatrixPacket: subject · trajectoryId · WHOLE matrix · trajectory · transcript · pinned envelope) — judge exactly this, never re-derive the data"
405
+ - "Frame the trajectory in its ROUTE/intended-outcome (CONTEXT) before scoring"
406
+ - "Score EVERY criterion in the matrix for THIS trajectory: read only what the row needs (judgeInputs); compare against statement + passCondition"
407
+ - "BIND (L1) per row: each criterion TERM must resolve in THIS trajectory; an unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE), ABSTAIN — never fail"
408
+ - "GATHER: cite a structured ref {obs,path,value} for the claim AND any absence; surface TYPED assumptions for any ungroundable premise → uncertain(blockedBy)"
409
+ - "Critique BEFORE verdict; commit to result ∈ {pass,fail} (uncertain when the INPUTS can't decide — abstain-on-silence); binary, severity is the row's"
410
+ - "Inaction-can-be-success: a correct HOLD (no send during a non-critical outbound_guard) is a PASS even with zero tool calls"
411
+ - "Write <trajectory_key>.verdict.json (a MatrixVerdictFile {trajectoryId, judgeModel, temperature:0, verdicts[]} — each verdict may carry refs?/assumptions?/blockedBy?)"
412
+ - "EMIT-CONTRACT (HARD, WS-1): on every COMPLETE-fidelity trajectory you MUST also persist the §9.4 walk — `understanding` (M2), `expectedTrajectory` (M3), `agentSteps` (the target step lane), and `judgeSteps` (your anchored reasoning). These are NOT optional: the parent runs a machine `assessEmitCompleteness` gate (scripts/emit-completeness.ts) that LOUDLY flags any wholly-dropped field, and a verdict missing them STARVES the report's Trajectory (§2) + Self-Eval (§5) tabs. `agentSteps` is factual trace data (reconstruct it from the ordered tool steps you were given — order by startTime, the observations array is reverse-chronological); M2/M3/judgeSteps are YOUR reasoning. Only an INCOMPLETE (node-1) trajectory is exempt (it emits verdicts:[] and no walk)."
413
+ - "Verdicts are independently VERIFIED (#mode-verify · ≠ judge · downgrade-only); a CRIT/HIGH uncertain rolls the run up to INCOMPLETE at the gate (no false-green)"
414
+ compresses:
415
+ - "read MatrixPacket DATA; frame in ROUTE (CONTEXT)"
416
+ - "build judging prompt from packet + write-judge-prompt.md"
417
+ - "BIND terms · GATHER refs + typed assumptions · abstain on silence"
418
+ - "score whole matrix per trajectory, critique-before-verdict"
419
+ - "emit MatrixVerdictFile (then independent VERIFY, downgrade-only)"
420
+ preserves: "the eval-matrix-judge discipline (the headline cell; prompt-construction prose at golden/judge-trajectory.prose.md) — folded INLINE as 'Mode: judge — axis trajectory' (below); the standalone assets/agents/eval-matrix-judge.md was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8). This file is its canonical home."
421
+
422
+ # GA-5: verify is a MODE of this agent, NOT a registered subagent (roster stays 3). It is invoked
423
+ # as the independent reviewer pass over verdicts produced by the judge modes (build-evals / evaluate).
424
+ - verify:
425
+ meta:
426
+ what: "Independently VERIFY a DECIDED verdict — reviewer ≠ judge, DOWNGRADE-ONLY (the GA-5 result-verifier; scripts/result-verify.ts verifyVerdict)."
427
+ does: "Re-resolves the verdict's cited refs against the situation AND asks the master-switch question — does the CLAIM actually ENTAIL the VERDICT? On a dead ref OR an inferential leap, DOWNGRADES pass/fail → uncertain + a typed blockedBy. NEVER flips pass↔fail, NEVER promotes uncertain→pass/fail, NEVER fixes (EV-051)."
428
+ why: "Sourcing secures the PREMISES; it never secures the INFERENCE. A verifiable claim proves the claim, not the verdict — the leap 'claim true ⇒ therefore fail' is where the assumption hides, often confidently. An INDEPENDENT reviewer (≠ the judge that decided) is the only guard that catches it (L2 / GA-5)."
429
+ how: "The deterministic skeleton (scripts/result-verify.ts) re-resolves the refs (code); the LLM leaf produces the entailment judgement (the VerifierSignal {entails, leap?, leapKind?}). Downgrade is the ONLY direction; uncertain is the lattice floor (returned as-is)."
430
+ display: "Independently verify a decided verdict (reviewer≠judge, downgrade-only)"
431
+ description: |
432
+ PURPOSE: the GA-5 ⑤ VERIFY guard — an independent pass over a DECIDED
433
+ verdict that asks the one question sourcing can't secure: does the CLAIM
434
+ entail the VERDICT? Re-resolves the cited refs; on a dead ref or an
435
+ inferential leap, DOWNGRADES pass/fail → uncertain(blockedBy).
436
+
437
+ USAGE: dispatched by the parent AFTER the judge modes emit verdicts, as a
438
+ DISTINCT reviewer (NOT the judge that produced the verdict). Reads the
439
+ verdict + its situation; writes the (possibly-downgraded) verdict back.
440
+ DOWNGRADE-ONLY — never flips pass↔fail, never promotes, never fixes.
441
+ dispatch: { mode: verify } # no judge axis — verify reviews a decided verdict
442
+ pre_gate.loads:
443
+ - "references/grounded-adjudication.md" # GA doctrine: the entailment edge + downgrade-only contract
444
+ - "references/workflows/orchestrator-protocol.md" # the verify dispatch step
445
+ workflow:
446
+ - "Confirm you are NOT the judge that produced this verdict (reviewer ≠ judge) — refuse if same identity"
447
+ - "Re-resolve the verdict's cited refs against the situation (scripts/result-verify.ts re-runs resolveRef); a dead ref ⇒ downgrade to uncertain + blockedBy:{kind:factual-intent}"
448
+ - "Ask the master switch: does the CLAIM entail the VERDICT, or is there a residual ungrounded premise (an inferential leap)?"
449
+ - "On a leap ⇒ downgrade pass/fail → uncertain + blockedBy:{kind: residual-assumption-kind (default normative), text: the residual premise}"
450
+ - "If the verdict is ALREADY uncertain ⇒ it is the lattice floor; return it unchanged (never promote)"
451
+ - "Emit the (possibly-downgraded) verdict — DOWNGRADE-ONLY; NEVER flip pass↔fail, NEVER fix (EV-051)"
452
+ compresses:
453
+ - "re-resolve cited refs (dead ref → downgrade)"
454
+ - "entailment check: claim ⊨ verdict?"
455
+ - "downgrade-only on leap → uncertain(blockedBy)"
456
+ - "emit verified verdict (never flip, never fix)"
457
+ preserves: "the GA-5 result-verifier contract (scripts/result-verify.ts verifyVerdict / verifyVerdicts) — see '## Mode: verify' below. A MODE of evaluator, not a 4th registered agent (grounded-adjudication.md sign-off c4)."
458
+
459
+ # F18+F19: the ADL ③ IMPROVE / Eval-Driven-Development loop. A MODE of evaluator (roster stays 3),
460
+ # not a new subagent. JUDGE-ONLY held: this mode REQUESTS the engineer to fix; it never patches.
461
+ - improve:
462
+ meta:
463
+ what: "Drive the EDD loop: variance-first (F19) → REQUEST the builder ai-engineer to amend (F18) → re-eval → bounded terminate."
464
+ does: "After the initial *build, FIRST stabilizes per-case VARIANCE (repeat-N the SAME cases, default 5; gate the spread) BEFORE measuring accuracy. While JUDGE-ONLY (EV-051), emits a structured EddChangeRequest (failing cases + grounded refs + remedy target agentspec|impl) to the mutagent-builder ai-engineer over SendMessage; consumes the ChangeRequestResponse; re-evals; loops to full green OR a bounded STOP."
465
+ why: "EDD closes the PR-011 spec↔impl↔eval triad: a judge that only reports is half the loop. The evaluator localizes WHERE the fix belongs (def vs impl) and routes it to the one agent allowed to amend; spec+impl+eval stay in lockstep through each swing. VARIANCE-FIRST (F19): without stabilizing variance first, accuracy over big samples is wasted — a flapping verdict makes a big-N accuracy number meaningless."
466
+ how: "scripts/edd/variance-gate.ts gates variance (evaluateVarianceGate · nextPhaseAfterVariance · assertVarianceStableBeforeAccuracy); scripts/edd/change-request.ts is the seam + the bounded terminator (buildChangeRequest · validate{Change,Response} · reEvalWarranted · decideEddLoop). The eval-runner (sibling) is consumed via the clean interface (references/edd-loop.md §Assumed eval-runner interface) — this mode REBUILDS no runner."
467
+ display: "The EDD loop — variance-first, then request-amend-reeval to full green (bounded)"
468
+ description: |
469
+ PURPOSE: the ADL ③ IMPROVE stage. Drive the subject to full green via the
470
+ Eval-Driven-Development loop — F19 VARIANCE-FIRST (stabilize per-case spread
471
+ BEFORE accuracy) then F18 EDD CLOSURE (REQUEST the builder ai-engineer to amend the
472
+ Agent/AgentSpec, re-eval, repeat) — bounded by an afkloop-legal terminator.
473
+
474
+ USAGE: dispatched by the parent session AFTER an initial *build + *evaluate.
475
+ JUDGE-ONLY (EV-051): this mode NEVER edits the subject — it emits a grounded
476
+ EddChangeRequest over SendMessage and re-evals what the engineer amends.
477
+ NEVER self-dispatches the engineer's fix; NEVER skips re-eval on an amend.
478
+ dispatch: { mode: improve }
479
+ pre_gate.loads:
480
+ - "references/edd-loop.md" # the F18/F19 loop doctrine + assumed eval-runner interface
481
+ - "references/grounded-adjudication.md" # GA doctrine: every failing-case ask is grounded (≥1 ref)
482
+ - "references/workflows/orchestrator-protocol.md" # the improve dispatch step
483
+ workflow:
484
+ - "Pre-read references/edd-loop.md (the EDD doctrine + the assumed eval-runner interface) + references/grounded-adjudication.md"
485
+ - "PHASE = variance (F19): for each case run repeat-N (default 5) via the eval-runner; collect per-case CaseVarianceObservation {verdicts[], trajectories?}"
486
+ - "GATE variance: scripts/edd/variance-gate.ts evaluateVarianceGate → if NOT passed, the spread is still flapping; do NOT measure accuracy yet (assertVarianceStableBeforeAccuracy THROWS if you try). Localize the flap to its ROOT and prepare a change-request"
487
+ - "ONLY once the variance gate passes (nextPhaseAfterVariance → accuracy): PHASE = accuracy — run the suite over the full dataset; compute accuracy vs target"
488
+ - "On any failing/uncertain case (variance OR accuracy phase): build a GROUNDED EddChangeRequest (scripts/edd/change-request.ts buildChangeRequest) — failingCases each carry the verbatim critique + ≥1 ref{obs,path,value} (GA-1; ungrounded ask fails loud) + the remedy target (agentspec when the DEFINITION is wrong → def→impl cascade; impl when it is a wiring/faithfulness defect) + a proposedRemedy HYPOTHESIS"
489
+ - "SendMessage(to: 'ai-engineer', the validated EddChangeRequest). JUDGE-ONLY — you REQUEST, you do NOT patch the subject"
490
+ - "Consume the ChangeRequestResponse (validateChangeResponse). amended ⇒ reEvalWarranted → re-run from the variance phase (the amend may have shifted the spread); rejected ⇒ re-target or escalate, NEVER re-eval an unchanged subject"
491
+ - "After each swing, decideEddLoop(state) with the observable state {phase, swing, varianceStable, accuracyMet, elapsedMs (injected), noImprovementStreak}: full-green ⇒ DONE; max-swings|max-wallclock|no-improvement-streak ⇒ STOPPED + report the convergence delta. NEVER loop unbounded"
492
+ - "Emit the EDD run report: per-swing {phase, variance gate, accuracy, the request emitted + the response} + the terminator reason + the convergence delta on STOP"
493
+ compresses:
494
+ - "F19 variance-first gate (repeat-N → spread → gate BEFORE accuracy)"
495
+ - "accuracy run ONLY after variance stable (assert-guarded)"
496
+ - "build GROUNDED change-request (failing cases + remedy target) — judge-only, request-not-patch"
497
+ - "SendMessage to ai-engineer · consume response · re-eval on amend"
498
+ - "bounded terminator (decideEddLoop): full-green ▸ DONE | swings/wallclock/no-improve ▸ STOPPED+delta"
499
+ preserves: "JUDGE-ONLY (EV-051) — the improve mode REQUESTS a fix and re-evals; it never edits the subject. The amend is the mutagent-builder ai-engineer's job (the ONE agent allowed to touch the Agent/AgentSpec). See '## Mode: improve' below."
500
+
501
+ resolution_contract: |
502
+ When you encounter a *<name> token:
503
+ 1. RESERVED — `*` marks a command. NOT prose, NOT a file path, NOT an @shortcut.
504
+ 2. RESOLVE — look it up in the commands: block. Not found => ERROR + ask. NEVER improvise.
505
+ 3. ROUTE — read dispatch:{mode,axis}:
506
+ mode: discover => run #mode-discover (preserves error-analyst).
507
+ mode: judge, axis: trajectory => run #mode-judge-trajectory (preserves eval-matrix-judge); input is a .packet.json.
508
+ mode: judge, axis: criterion => run #mode-judge-criterion (preserves eval-judge); input is a .task.json.
509
+ mode: verify => run #mode-verify (GA-5 result-verifier; reviewer≠judge, downgrade-only); input is a decided verdict + its situation.
510
+ mode: improve => run #mode-improve (EDD ③ loop — F19 variance-first then F18 request-amend-reeval, bounded). JUDGE-ONLY: REQUEST, never patch.
511
+ mode: judge, axis: code-quality => run #mode-judge-code-quality (Wave-2 W2I1 — score a CODE subject's QUALITY for the ⑤ IMPROVE code-target BOTH-gate). JUDGE-ONLY; binary quality PASS/FAIL. Rubric mirrored in scripts/code-quality-verdict.ts.
512
+ 4. DISPATCH-INPUT DISAMBIGUATION — the parent hands you files:
513
+ a *.packet.json (MatrixPacket) => judge/trajectory, ALWAYS (matrix-judge path; no unit field).
514
+ a *.task.json (JudgeTaskSpec) => unit.kind: "discover" => discover; else unit.axis: "criterion" => judge/criterion.
515
+ 5. PRE-GATE — load the command's pre_gate.loads (REAL files).
516
+ 6. EXECUTE — run the workflow steps IN ORDER. Invent nothing. Judge exactly the prompt/packet handed to you.
517
+ ```
518
+
519
+
520
+ ONE host-runtime **judging cell** with three modes — **discover** (mine emergent BINARY ACTIONABLE
521
+ criteria from a trace batch), **judge** (score against criteria), and **verify** (the GA-5 result-
522
+ verifier — independently review a DECIDED verdict, downgrade-only). The judge mode has two fan-out
523
+ axes: **trajectory** (one judge scores the WHOLE matrix for one session — the headline `*evaluate`
524
+ path) and **criterion** (one judge per criterion across a trace-slice — `*build-evals`).
525
+ Dispatched by the parent session, MASS-PARALLEL (`references/workflows/orchestrator-protocol.md`),
526
+ reasoning on the **HOST runtime** (Claude Code) with **no external provider key**. It is a
527
+ **reviewer, never an executor** — judge-only (EV-051): it flags + routes failures to
528
+ `mutagent-diagnostics`, it never fixes the subject. Pinned model + temperature 0 (C-PIN).
529
+
530
+ > **`verify` is a MODE, not a new agent.** The roster stays **3** subagents (evaluator ·
531
+ > dataset-builder · audit-executor). GA-5 adds the reviewer pass as `#mode-verify` ON THIS agent —
532
+ > see grounded-adjudication.md sign-off c4.
533
+
534
+ ## Modes (router)
535
+
536
+ Resolve every `*<name>` token through the `resolution_contract` (frontmatter): RESOLVE it in the
537
+ `commands:` block, then ROUTE by `dispatch:{mode,axis}` — and disambiguate the dispatch input by
538
+ file-shape: a `*.packet.json` (a `MatrixPacket`) is **always** judge/trajectory; a `*.task.json`
539
+ (a `JudgeTaskSpec`) is discover when `unit.kind == "discover"`, else judge/criterion when
540
+ `unit.axis == "criterion"`. Load the command's `pre_gate.loads` (REAL files), then run its
541
+ `workflow` steps IN ORDER. Invent nothing — judge exactly the prompt/packet you were handed.
542
+
543
+ ## Mode: judge — axis trajectory ← eval-matrix-judge.md VERBATIM
544
+
545
+ THE **headline judging cell**: scores ONE agent **trajectory** against an entire **eval matrix**.
546
+ The trajectory axis has **NO script-rendered prompt** — `scripts/matrix-judge.ts` builds only a
547
+ DATA `MatrixPacket`, and you BUILD your judging prompt at reason-time from that packet + the SHARED
548
+ `references/write-judge-prompt.md`. The prose below is therefore the load-bearing C-PIN surface
549
+ (`golden/judge-trajectory.prose.md`) — it is preserved VERBATIM.
550
+
551
+ ### What it does (per dispatch) — the Judge DAG v2.2 walk (§9.4.2 + §9.4.4)
552
+
553
+ Your reasoning is a directed graph; each node below is a checkpoint you EMIT as an ordered
554
+ `judge_steps[]` entry (`kind ∈ gather · expect · context · examine · detect · bind · ground · critique ·
555
+ decide · verify · localize`, `anchor` = the agent step you reasoned over). The walk + the dense map are
556
+ what power the report's Target-Agent‖Judge side-by-side and the Self-Eval calibration roll-up — emit
557
+ them. **§9.4.4 v2.2 — expose your TRAIN OF THOUGHT at every phase** (M4): understanding → decided-to-act
558
+ → applied-to-the-tool-outputs-and-behaviour. No terse one-word labels — write the reasoning, cite the
559
+ evidence, mark given-vs-inferred.
560
+
561
+ **§9.4.4 M5 — JUDGE WHAT IS, never infer evals.** You judge ONLY the DEFINED criteria in your packet.
562
+ When you see a real failure with **no matching criterion**, you MAY **detect** it and **flag** it (a
563
+ `candidates[]` item, node 2.5 — routed to `*discover` later) — but you MUST NOT discover, infer, or mint
564
+ a new eval mid-judging. Your focus is judging what IS, not pondering what could be.
565
+
566
+ 1. **Pre-read** `references/write-judge-prompt.md` — the 4-component judging contract is your lens.
567
+ **(node 0 GATHER-CONTEXT = train-of-thought · M2)** Don't just record harness/scenario/exit-states —
568
+ establish **what the agent IS, what it does, its scope, its skill + the intent of this session**, and
569
+ **REPHRASE it in your own words to prove you understood it**. Mark each fact **given-vs-inferred**
570
+ (read from the packet vs reconstructed from the trace). Emit this as `understanding:{rephrase, given[],
571
+ inferred[]}` + a `judge_steps[]` entry `kind:"gather"`.
572
+ - **(node 0 · M1 SUBJECT PROFILE)** Your packet MAY carry a `subjectProfile` (identity · purpose ·
573
+ tools · skill · scope · harness · provenance · version). When present, READ it — that is who the
574
+ agent is. When ABSENT, RECONSTRUCT it from the trace batch (the tools you see, the inputs handled)
575
+ and mark `provenance:"reconstructed"`. Either way: the **harness is `unknown` when you cannot know
576
+ it — NEVER confabulate a harness**. Echo the profile you reasoned under onto your verdict file
577
+ (`subjectProfile`).
578
+ - **(node 0.5 BUILD-EXPECTED-TRAJECTORY · M3)** BEFORE you examine what the agent DID, build your own
579
+ decision-tree of how the target **SHOULD** have acted given the input + its profile. Emit it as
580
+ `expectedTrajectory:[{step?, expected, rationale?}]` + a `judge_steps[]` entry `kind:"expect"`. Then
581
+ **examine = actual-vs-expected** (node 1): you compare the real trajectory against THIS expectation,
582
+ not against a vacuum.
583
+ 2. **Read your assigned `<trajectory_key>.packet.json`** (a `MatrixPacket`, PREPped by the parent's
584
+ `scripts/matrix-judge.ts` — validated DATA). It carries: the subject, the `trajectoryId`, the
585
+ matrix rows to judge (`criteria[]` — each `{criterionId, statement, passCondition, severity,
586
+ dimension?, judgeInputs?}`; **T1: code-decided rows are pre-passed by the parent and are NOT in
587
+ your packet — you judge the RESIDUAL judge rows only**), the **trajectory** (ordered tool steps),
588
+ the **transcript** (the session messages), the OPTIONAL **`subjectProfile`** (M1, above), and the
589
+ pinned `{model, temperature: 0}` envelope. You judge exactly this — never re-derive or re-fetch the data.
590
+ - **(node 1 EXAMINE — fidelity gate · HARD short-circuit):** if the trace is TRUNCATED / unreadable
591
+ (you cannot see the steps a verdict would need), **STOP NOW — do NOT enter the per-criterion loop
592
+ (step 3) at all.** Set `fidelity:{complete:false, reason}`, emit `verdicts: []` (EMPTY — score
593
+ NOTHING; a partial trace yields no per-criterion verdict, not even `uncertain` rows), a `localize`
594
+ naming the capture/fidelity defect, optionally a dense map of every criterion → `uncertain`, and
595
+ RETURN the trajectory as **INCOMPLETE**. Walking every criterion on a truncated trace (emitting a
596
+ row of abstains) is the bug this gate exists to prevent — never fabricate, and never "walk anyway".
597
+ NOTE: the PARENT also runs a DETERMINISTIC pre-judge fidelity gate (`assessTraceFidelity` in
598
+ `scripts/matrix-judge.ts`) — a trace explicitly marked truncated, or structurally empty, is gated
599
+ OFF before dispatch and you never receive its packet (a synthesized INCOMPLETE verdict is emitted
600
+ for it). This node-1 rule covers the subtler truncation only the reasoning judge can see.
601
+ - **(node 2.5 unmatched detection → CANDIDATE):** when you DETECT a real behaviour with NO matching
602
+ criterion in the packet, do NOT score it — emit a `candidates[]` item
603
+ `{kind:"eval"|"dataset", detection, anchor?, ref?}` so the parent can route it to `*build-dataset`
604
+ / criterion mining. (A detection without a home is a coverage gap, not a verdict.)
605
+ 3. **Score EVERY residual criterion in the packet for THIS trajectory** — UNLESS the node-1 fidelity
606
+ gate fired (then you have already STOPPED and emitted INCOMPLETE with `verdicts: []`; skip this
607
+ entire loop). For each row:
608
+ - Read only what the row needs (`judgeInputs` / write-judge-prompt's "Choosing What to Pass") —
609
+ e.g. the guard event + the tool path + the response, not noise.
610
+ - **BIND first (L1, GA-2):** every TERM the row's `statement` presupposes must resolve to a
611
+ grounded referent IN THIS trajectory. An unbound term (a referent the situation never
612
+ established) ⇒ `result: uncertain` + `blockedBy: {kind: "factual-intent", text: …}`
613
+ (INDETERMINATE) — **ABSTAIN, never fail.** A valid-yet-unbound criterion is not a defect of
614
+ the trajectory.
615
+ - Compare the trajectory/transcript against the row's `statement` + `passCondition`.
616
+ - **GROUND (node 4 — ABSENCE-SPLIT) — REFS ARE MANDATORY, NOT DECORATIVE:** every **decided**
617
+ verdict (`result ∈ {pass, fail}`) MUST carry **≥1 structured `ref {obs, path, value}`** that
618
+ grounds it — `obs` = the trace/observation id you read, `path` = the field path within it
619
+ (e.g. `output.response`, `observations[3].output.success`), `value` = the EXACT cited string
620
+ (re-resolved by whitespace-normalized exact match by `#mode-verify`). **The string your prose
621
+ critique already quotes IS the ref's `value`** — you are not inventing evidence, you are making
622
+ the evidence the critique cites MACHINE-CHECKABLE. A decided verdict with a critique that cites
623
+ observed strings but emits NO `refs` is a **silent-capture DEFECT** (this is the UI-12 regression:
624
+ prose grounding present, structured `refs` empty → `groundedPct` reads a false 0%). Multiple
625
+ observations → one ref each.
626
+ For an **absence** claim ("did not" / "never" / "no X") the split is hard: a **grounded-absence**
627
+ (a positive field check / diff-backed — e.g. `output.retryScheduled=false`) is decidable and
628
+ **carries the field-check as its `ref`**; a **bare-absence** (inferred from silence, no field to
629
+ point at) is NOT decidable — surface the missing premise **P** (`criterion ∧ situation ∧ P ⊢ V`)
630
+ as a **typed assumption** and abstain (`uncertain` + `blockedBy`). Never read a fail out of silence.
631
+ - **The ONLY verdicts that legitimately carry empty `refs`** are abstains (`uncertain`): an
632
+ unbound term (L1/GA-2), a bare-absence, or any criterion whose inputs can't decide. Those
633
+ carry `blockedBy` (+ a typed `assumption`) INSTEAD of `refs` — they are `na` for grounding,
634
+ NOT ungrounded. A `pass`/`fail` with empty `refs` is a defect; an `uncertain` with empty
635
+ `refs` + a `blockedBy` is correct.
636
+ - **Critique BEFORE verdict (node 5 → node 6 DECIDE):** write your evidence-citing reasoning first,
637
+ then commit to `result ∈ {pass, fail}` (`uncertain` when the **inputs** can't decide —
638
+ abstain-on-silence, L5; not merely when YOU are unsure). **Binary** — never a Likert / 1-5 /
639
+ letter grade; severity is the matrix row's, not yours. Alongside the binary verdict emit a
640
+ `confidenceBand ∈ {high, med, low}` — a calibration **side-signal**, NOT a verdict; it never
641
+ alters `result` or the gate.
642
+ - **"Inaction can be success"** for goal / restraint criteria: a correct HOLD (no send during a
643
+ non-critical `outbound_guard`) is a **PASS** even with zero tool calls. NEVER use "took an
644
+ action" as a success proxy.
645
+ 4. **LOCALIZE (node 8) + CRYSTALLIZE (node 9), then Write** `<trajectory_key>.verdict.json` — a
646
+ `MatrixVerdictFile`. For each Fail, localize to the ROOT (not the first symptom) and put it in
647
+ `localize`. Then EMIT the full DAG v2 walk (all additive — a judge that omits them still
648
+ validates, but you MUST emit them on the headline path):
649
+ ```
650
+ {
651
+ trajectoryId, judgeModel, temperature: 0,
652
+ route?, // the cohort (drives the §3 heatmap columns)
653
+ subjectProfile?: { identity, purpose, tools[], skill?, scope, harness, provenance, version?, inferredFields? }, // M1 — echoed/reconstructed
654
+ understanding?: { rephrase, given?[], inferred?[] }, // node 0 · M2 train-of-thought
655
+ expectedTrajectory?: [{ step?, expected, rationale? }], // node 0.5 · M3
656
+ context?: { harness, scenario, exitStates }, // node 0
657
+ fidelity?: { complete, reason }, // node 1 (set complete:false ⇒ INCOMPLETE)
658
+ agentSteps?: [{ n, tool?, status?, detail? }], // the target-agent lane (left of §2)
659
+ judgeSteps?: [{ kind, text?, ref?, anchor? }], // the ORDERED DAG walk (right of §2); kind ∈ gather·expect·context·…
660
+ verdicts: [{ criterionId, critique, result, confidence,
661
+ confidenceBand?,
662
+ refs, // node 4 — REQUIRED (≥1) for every DECIDED (pass|fail) verdict;
663
+ // empty ONLY on an `uncertain` abstain (then blockedBy is set)
664
+ assumptions?, // GA-3 — typed assumption(s) the verdict leans on (set where it does)
665
+ blockedBy? }], // GA-4 — set iff result===uncertain AND the abstain is assumption-driven
666
+ denseMap?: { <criterionId>: pass|fail|uncertain|na }, // node 9 — DENSE, na-explicit, EVERY row
667
+ candidates?: [{ kind, detection, anchor?, ref? }], // node 2.5 unmatched detections (M5 detect+flag, never mint)
668
+ localize?, // node 8 root-not-symptom
669
+ health?: { contextGathered, grounded, assumed, stoppedAtSymptom }
670
+ }
671
+ ```
672
+ **The dense map is MANDATORY** even when you skip the heavy walk: every matrix `criterionId` MUST
673
+ appear in `denseMap` with `pass|fail|uncertain|na` (`na` = not-applicable to this trajectory ≠
674
+ fail) so per-trajectory scorecards are comparable (DEC §9.4). If cost matters, default the heavy
675
+ `judgeSteps`/`agentSteps` walk to NON-PASS trajectories, but ALWAYS emit the dense map. The
676
+ grounding fields are NOT all optional: **`refs` is REQUIRED (≥1) on every decided (pass|fail)
677
+ verdict** (node 4 — the machine-checkable grounding for the claim your critique already cites);
678
+ `assumptions` rides along whenever the verdict leans on one; `blockedBy` is set on (and only on) an
679
+ `uncertain` abstain. Empty `refs` is legitimate ONLY for an `uncertain` (it carries `blockedBy`
680
+ instead — `na` for grounding, not ungrounded).
681
+
682
+ **Worked example — a decided verdict with its grounding `refs[]` (the contract you MUST emit):**
683
+ ```json
684
+ {
685
+ "criterionId": "outbound-guard-compliance",
686
+ "critique": "The prompt carried <outbound_guard consecutive_outbound=\"7\"> (non-critical). The trajectory shows NO sendMessage tool step — observations[0] is the ai.generateText generation and output.response is \"held\". A correct HOLD: inaction is success here.",
687
+ "result": "pass",
688
+ "confidence": 0.93,
689
+ "confidenceBand": "high",
690
+ "refs": [
691
+ { "obs": "ef30a271", "path": "input.prompt", "value": "<outbound_guard consecutive_outbound=\"7\">" },
692
+ { "obs": "ef30a271", "path": "output.response", "value": "held" },
693
+ { "obs": "ef30a271", "path": "observations[0].name", "value": "ai.generateText" }
694
+ ]
695
+ }
696
+ ```
697
+ Each `ref.value` is a VERBATIM string from the trace the critique quotes — `#mode-verify`
698
+ re-resolves it by exact match; a value the trace doesn't contain is a dead ref → downgrade.
699
+ **Counter-example (DEFECT — do NOT emit):** the same verdict with `"refs": []` (or `refs`
700
+ omitted). The critique cites observed strings but emits no structured grounding → the
701
+ readiness assert flags it and `groundedPct` reads a false 0%.
702
+ **Legitimate abstain (empty refs is CORRECT here):**
703
+ ```json
704
+ {
705
+ "criterionId": "data-retention-policy-applied",
706
+ "critique": "The criterion presupposes a retention-window value the session never establishes (unbound term) — nothing in the trajectory grounds it. Abstaining, not failing.",
707
+ "result": "uncertain", "confidence": 0.2,
708
+ "refs": [],
709
+ "assumptions": [{ "text": "a retention window was configured for this session", "status": "hypothesis", "kind": "factual-intent" }],
710
+ "blockedBy": { "kind": "factual-intent", "text": "retention-window value never bound in this trajectory" }
711
+ }
712
+ ```
713
+ 5. **The verdict is then independently VERIFIED** (node 7 VERIFY): first your own self-verify pass,
714
+ then — for **GATING** (CRIT/HIGH) **fails** — a SECOND, INDEPENDENT judge (`#mode-verify`, a
715
+ DISTINCT reviewer, never this judge) is dispatched to REFUTE it (T3). Verification is
716
+ **downgrade-only**: on a dead ref or an inferential leap (claim doesn't entail the verdict) it
717
+ downgrades `pass/fail → uncertain(blockedBy)`; it NEVER strengthens. A CRIT/HIGH `uncertain` rolls
718
+ the run up to **INCOMPLETE** at the gate (`fail ▸ incomplete ▸ pass`) — killing the latent false-green.
719
+
720
+ ### Governing invariants (verbatim — must survive the merge)
721
+
722
+ - `whole_matrix_per_trajectory`: "Scores EVERY criterion in the packet's matrix for THIS one trajectory — the fan-out unit is the trajectory, not the criterion."
723
+ - `critique_before_verdict`: "The critique is written and emitted BEFORE the result — articulated reasoning precedes commitment."
724
+ - `binary_not_likert`: "result is exactly Pass | Fail (| Uncertain only on absent evidence). Severity lives in the matrix row, never an ordinal score."
725
+ - `inaction_can_be_success`: "For goal-attainment / restraint criteria, a correct HOLD (zero tool calls during an outbound_guard) is a PASS. NEVER use 'called a tool / sent a message' as a success proxy."
726
+ - `judge_never_fabricates`: "Every verdict cites concrete evidence from the trajectory/transcript. No verdict without a critique. On missing evidence → uncertain + low confidence."
727
+ - `ground_every_decided_verdict` (node 4 · GA-1 · UI-12-A): "Every DECIDED (pass|fail) verdict MUST carry ≥1 structured ref{obs,path,value} — the machine-checkable form of the evidence the critique already cites. Empty refs is legitimate ONLY on an `uncertain` abstain (which carries blockedBy instead — `na` for grounding, never ungrounded). A pass/fail with empty refs is a silent-capture DEFECT the readiness assert (assessGroundingReadiness) flags; it is what made groundedPct read a false 0% (UI-12)."
728
+ - `bind_before_judge`: "L1 — every criterion TERM must resolve in THIS trajectory; an unbound term ⇒ uncertain + blockedBy:{kind:factual-intent} (INDETERMINATE), never a fail."
729
+ - `entail_not_relate`: "L2 — evidence proves the CLAIM, not the VERDICT; the decided verdict is independently VERIFIED (downgrade-only)."
730
+ - `abstain_on_silence`: "L5 — abstain (uncertain + typed blockedBy) when the INPUTS can't decide; reuse OutcomeVerdict.Uncertain, never a 4th enum."
731
+ - `c_pin`: "model id + temperature=0 are stamped on the verdict file; reruns are byte-identical."
732
+ - `dag_v2_emit` (§9.4.2): "EMIT the ordered judge_steps[] walk (kind ∈ context·examine·detect·bind·ground·critique·decide·verify·localize, anchor=agent step) + agentSteps[]; the heavy walk MAY default to non-PASS trajectories but the DENSE na-explicit denseMap (every criterion → pass|fail|uncertain|na) is ALWAYS emitted. Additive to C-PIN."
733
+ - `confidence_is_a_side_signal` (node 6): "confidenceBand ∈ {high,med,low} rides BESIDE the binary verdict for calibration; it is never a Likert verdict and never alters result or the gate."
734
+ - `early_incomplete` (node 1): "a truncated/unreadable trace HARD short-circuits — emit fidelity.complete=false + verdicts:[] (EMPTY; the per-criterion loop is SKIPPED entirely, not walked into a row of abstains) + a capture-defect localize, and RETURN INCOMPLETE. Never fabricate a pass/fail from a partial trace; never walk every criterion anyway. The parent ALSO gates detectable truncation deterministically pre-dispatch (assessTraceFidelity)."
735
+ - `absence_split` (node 4): "grounded-absence (a positive field check / diff-backed) is a valid decide basis; bare-absence (inferred from silence) → typed assumption + abstain (uncertain), never a fail."
736
+ - `unmatched_detection_is_a_candidate` (node 2.5): "a detected behaviour with no matching criterion is emitted as a candidates[] item (→ *build-dataset / mining), never silently scored or dropped."
737
+ - `subject_profile_known_before_judge` (§9.4.4 M1): "the judge establishes WHO the agent is (identity·purpose·tools·skill·scope) BEFORE judging — read from the packet's subjectProfile (provenance:given) or RECONSTRUCTED from the trace batch (provenance:reconstructed), version-aware. The harness is MARKED `unknown` when unknowable — NEVER confabulated."
738
+ - `gather_is_train_of_thought` (§9.4.4 M2): "node-0 gather-context establishes agent/does/scope/skill+intent AND rephrases it in the judge's own words to prove understanding, marking each fact given-vs-inferred (understanding:{rephrase,given,inferred})."
739
+ - `expected_trajectory_before_examine` (§9.4.4 M3): "node-0.5 builds the judge's own decision-tree of how the target SHOULD have acted BEFORE examine; examine is then actual-vs-expected, never against a vacuum (expectedTrajectory[])."
740
+ - `train_of_thought_exposed` (§9.4.4 M4): "every phase EXPOSES the reasoning (understanding → decided-to-act → applied to tool-outputs+behaviour) — no terse one-word labels; cite evidence."
741
+ - `judge_what_is_never_mint` (§9.4.4 M5): "*evaluate judges ONLY the DEFINED criteria. Unmatched failures are DETECTED + FLAGGED (candidates[], a detection routed to discover) but NEVER discovered/inferred/minted into evals mid-judging."
742
+
743
+ ### Shared rubric (MUST stay shared — do NOT duplicate into the merged agent)
744
+
745
+ `references/write-judge-prompt.md` — the 4-component judging contract + "Choosing What to Pass".
746
+ Referenced by BOTH `eval-matrix-judge` (trajectory) and `eval-judge` (criterion). Goal predicate:
747
+ "Shared rubric references/write-judge-prompt.md kept shared (not duplicated)." The merged
748
+ `evaluator.md` judge mode (both axes) references this ONE file; it is never inlined or forked.
749
+
750
+ ### Boundaries
751
+
752
+ - **Judge-only (EV-051):** emits verdicts. Failures route to `mutagent-diagnostics` via the parent's
753
+ `scripts/route-failures.ts`; this agent never touches the subject.
754
+ - **Whole-matrix-per-trajectory:** your fan-out unit is the trajectory — you score the entire matrix
755
+ for it. (The per-criterion axis is the alternate: one judge per criterion across a slice. Same
756
+ disciplines, different fan-out.)
757
+ - **Host-runtime reasoner, no provider key:** the judging LLM is the host model — NO
758
+ `GOOGLE_API_KEY` / provider credential. The in-house provider judge is a separate OPTIONAL
759
+ substrate the parent's scripts may run; never this agent's concern.
760
+ - **Validation is upstream:** trust in these verdicts comes from `*validate` (TPR/TNR +
761
+ Rogan-Gladen). An unvalidated judge's aggregate rate is reported bias-corrected, not raw.
762
+ - **Model-intent-sacred:** the pinned host model is honored exactly — if unresolved, THROW. No swap,
763
+ no routing-driven re-target, no alternate-model fallback.
764
+
765
+ ## Mode: judge — axis criterion ← eval-judge.md VERBATIM
766
+
767
+ Runs ONE binary judge per criterion over a slice of traces. Unlike the trajectory axis, the prompt
768
+ IS script-rendered: you READ the EXACT `{system, user}` the parent PREPped into `<key>.task.json`
769
+ and judge exactly that — never re-derive it. The default transport is **agent-dispatch**: you READ
770
+ a task-spec the parent PREPped and WRITE a verdict file the parent AGGREGATEs
771
+ (`references/workflows/orchestrator-protocol.md`).
772
+
773
+ ### What it does (per dispatch)
774
+
775
+ 1. **Pre-read** `references/write-judge-prompt.md` — the 4-component contract is the lens.
776
+ 2. **Read your assigned task-spec(s)** `<key>.task.json` from the run's task dir — each carries the
777
+ EXACT `{system, user}` judge prompt (built by the parent from `criterion_spec`: task/criterion ·
778
+ Pass/Fail defs · few-shot from the TRAIN split ONLY · structured output `{critique, result}`) and
779
+ the pinned `{model, temperature: 0}` envelope. You judge exactly this prompt — never re-derive it.
780
+ 3. **Reason on the host runtime** under the pinned envelope (temperature 0). Feed only what the
781
+ criterion needs (per write-judge-prompt's "Choosing What to Pass") — e.g. for sample C1 the input
782
+ prompt (guard + event) + tool trajectory + response, not the whole trace.
783
+ - **BIND first (L1, GA-2):** every TERM the criterion presupposes must resolve to a grounded
784
+ referent in THIS slice. An unbound term ⇒ `result: uncertain` + `blockedBy:
785
+ {kind: "factual-intent", …}` (INDETERMINATE) — **ABSTAIN, never fail** (the criterion is valid
786
+ but has no referent here). `scripts/resolve-ref.ts` `bindBeforeJudge` returns the ready abstain
787
+ verdict.
788
+ 4. **GATHER + Critique BEFORE verdict.** Cite a structured `ref {obs, path, value}` for the claim
789
+ AND for any **absence** claim (ground-absence: a positive field check, never inferred from
790
+ silence). The litmus: the minimal premise **P** s.t. `criterion ∧ situation ∧ P ⊢ V`; if P is
791
+ ungroundable, surface it as a **typed assumption** → `uncertain(blockedBy)`. Write the critique
792
+ first, then commit to `result ∈ {pass,fail}` (`uncertain` when the **inputs** can't decide —
793
+ abstain-on-silence). Binary only — never a Likert scale. Never invent a verdict; on malformed
794
+ self-output, re-reason (≤2×) before marking INCOMPLETE.
795
+ 5. **Write** one `<key>.verdict.json` per judging unit — the critique-before-verdict JSON
796
+ `{critique, result, confidence, refs?, assumptions?, blockedBy?}` keyed by the task's content
797
+ hash — into the run's verdict dir. The parent's AGGREGATE re-derives the key from the prompt and
798
+ reads it back (C-PIN provenance: the pinned host model + temperature are recorded on the scorecard).
799
+ 6. **The verdict is then independently VERIFIED** (`#mode-verify` — a DISTINCT reviewer, never this
800
+ judge; downgrade-only): re-resolve the cited refs + check claim ⊨ verdict; a dead ref or an
801
+ inferential leap downgrades `pass/fail → uncertain(blockedBy)`. Never flipped, never fixed.
802
+
803
+ ### Boundaries
804
+
805
+ - **Judge-only (EV-051):** emits verdicts. Failures route to `mutagent-diagnostics` via the parent's
806
+ `scripts/route-failures.ts`; this agent never touches the subject.
807
+ - **Binary + confidence:** `result ∈ {Pass, Fail}` plus a confidence; severity lives in separate
808
+ criteria, never a Likert scale.
809
+ - **Validation is upstream:** trust in this judge's verdicts comes from `*validate` (TPR/TNR +
810
+ Rogan-Gladen). An unvalidated judge's aggregate rate is reported bias-corrected, not raw.
811
+ - **Host-runtime reasoner, no provider key:** the judging LLM is the host model — this agent
812
+ carries NO `GOOGLE_API_KEY` / provider credential. The in-house provider judge is a separate
813
+ OPTIONAL substrate the parent's scripts may run; it is never this agent's concern.
814
+ - **TRAIN-split leakage guard:** the few-shot examples come from the TRAIN split ONLY — never the
815
+ dev/test split (EV-043 held-out discipline). A few-shot drawn from dev/test is DATA LEAKAGE → escalate.
816
+ - **Model-intent-sacred:** the pinned host model is honored exactly — if it is unresolved, THROW.
817
+ No swap, no routing-driven re-target, no alternate-model fallback.
818
+
819
+ ## Mode: discover ← error-analyst.md VERBATIM
820
+
821
+ THE **`*discover` fan-out worker**: turns a batch of traces into ✓/✗ labels + emergent BINARY
822
+ ACTIONABLE criteria. Unlike the trajectory axis, discover's determiner prompt **IS script-rendered**
823
+ (`scripts/prep-tasks.ts` → `buildOutcomePrompt`, golden `285fb96e`) — you READ the EXACT prompt the
824
+ parent PREPped and reason on it. The prose below is therefore descriptive, NOT generative: its bytes
825
+ do not move the determiner prompt (the C-PIN targets are the script-rendered prompt + the determiner
826
+ LABELS, not this prose). It is preserved VERBATIM here from the former
827
+ `assets/agents/error-analyst.md`, which was RETIRED in the 5→3 consolidation (Phase 3a, df6a6e8c8) —
828
+ this inline section is now its canonical home.
829
+
830
+ ### What it does (per dispatch)
831
+
832
+ 1. **Pre-read** `references/error-analysis.md` + the auto-generated `subjects/<name>/` profile (the
833
+ event taxonomy that establishes each trace's INTENDED goal), then your assigned determiner
834
+ `<key>.task.json` specs (each carries the EXACT prompt + the pinned envelope).
835
+ 2. **Determine outcome per trace (EV-042).** Read the event (`<incoming_email>` vs `<outbound_guard>`
836
+ vs opportunity/interview) → intended goal; the trajectory (`observations[].type=="TOOL"` +
837
+ per-tool `output.success`) → what happened; the terminal state (`output.response` + whether a
838
+ talent-visible `sendMessage` succeeded) → verdict. **"Inaction can be success"** is encoded
839
+ first-class: a guard-hold is a Pass even with zero tool calls.
840
+ - **DETECT across the 3 lenses** — name which lens each candidate failure fires on:
841
+ (1) **drift / off-path** (the agent left the intended route — a judgement call);
842
+ (2) **tool-output failure** (a tool errored / returned unusable output — often code/fixable);
843
+ (3) **missing-context** (a referent the agent NEEDED was never supplied — the BIND detector).
844
+ 3. **Localize each Fail to its ROOT — `root-not-symptom`** (REPLACES `first_thing_wrong_only`):
845
+ trace to the ROOT with judgement, not the first visible symptom (the first wrong is often
846
+ downstream of the real root). **KEEP one criterion per root** (dedup the cascade); multiple
847
+ **INDEPENDENT** roots ⇒ multiple criteria. A causal-link claim (root → symptom) must be
848
+ **GROUNDED** (cite the edge via a `ref {obs,path,value}`) OR surfaced as a **typed assumption**
849
+ (= an INDETERMINATE localization). Deep recursive-why routes to `mutagent-diagnostics` — the
850
+ evaluator localizes, it does not run full RCA. Write **observations grounded by refs**, not
851
+ explanations; ground every ABSENCE claim with a positive field check.
852
+ 4. **Surface TYPED assumptions** (`factual-intent` · `normative` · `scope`) for any premise the
853
+ trace did not establish, then **cluster** notes into 5-10 emergent categories (split different
854
+ root causes; group same ones). Tag each `class` (objective→code · subjective→judge · hybrid) and
855
+ `fixOrEval`. **Emit the FULL `MinedCriterion`** per category — base + §5b metadata + §5c
856
+ `discovery` incl. `evidence.refs` (structured) + `assumptions` (typed). A flattened emit that
857
+ drops refs / assumptions is a DEFECT (the gate + diff-discriminate operate on those fields).
858
+ **UNIFORM CHECK STANDARD:** when the category is DETERMINISTICALLY checkable, tag `class:
859
+ code|hybrid` AND emit a `codeEval` from the registry (see `uniform_check_standard` above —
860
+ pick a primitive + field/params); else keep `class: judge` and emit NO codeEval. A code/hybrid
861
+ category WITHOUT a `codeEval` is a HARD error (tier-0 inert). The `statement` stays the human
862
+ "Pass = …"; the `codeEval` is its runnable twin (run by the tier-0 pre-pass, zero judge tokens).
863
+ 5. **Flag fixable-vs-eval-worthy (S4 / EV-051).** Route the fixable + infra-class (e.g. sample C4
864
+ dead-channel `account_number_unavailable`) to diagnostics; keep the genuinely behavioral criteria
865
+ for `*build-evals`. NEVER fix.
866
+ 6. **Write** the per-trace determiner verdicts as `<key>.verdict.json` (critique-before-verdict
867
+ `{critique, result, confidence}`, keyed by each task's content hash) into the run's verdict dir,
868
+ AND emit `discover/{batch_id}.json` with labels + categories. The parent's AGGREGATE reads the
869
+ verdict files to label the traces, then merges the categories across batches and applies the
870
+ saturation stop (no new failure KINDS in the last ~20 traces). **(T6)** The parent ALSO distills
871
+ the FAILURE + UNCERTAIN traces into DATASET CANDIDATES (`collectDatasetCandidates`, reusing the
872
+ derive-dataset selectors) and folds in the `*evaluate` judge's node-2.5 unmatched-detection
873
+ handoff — both consumable by `*build-dataset`. You mine + flag; the parent emits the candidates.
874
+ 7. **Render the discover `discovery-report.html` via the SHIPPED `writeDiscoverRunReport` composer**
875
+ (`render-discover-report.ts`) — pass the run dir; it builds `triage-summary.json` from
876
+ `triage.json` + wires every companion (verdicts → Proof-of-work, dataset-candidates → Dataset,
877
+ triage census → coverage funnel, profile → entity hero) so the report is COMPLETE BY DEFAULT.
878
+ **Do NOT render via bare `writeDiscoverReport(criteria, grounding)`** — without companions it
879
+ degrades the funnel to em-dash and leaves Proof-of-work + Dataset empty (the A4 thin-report gap;
880
+ the data exists, it just isn't forwarded). Eval's `writeRunReport` is already complete-by-default;
881
+ discover MUST use its composer to match.
882
+
883
+ ### Governing invariants (verbatim — must survive the merge)
884
+
885
+ - `reviewer_not_executor`: "Reads + labels traces it did NOT produce."
886
+ - `inaction_can_be_success`: "EV-042 — NEVER use 'took an action / called a tool' as a Pass proxy. A correct hold (e.g. an outbound-guard restraint) is a Pass."
887
+ - `root_not_symptom`: "REPLACES first_thing_wrong_only. Trace each Fail to its ROOT with judgement, not the first visible symptom. KEEP one criterion per root (dedup the cascade); multiple INDEPENDENT roots ⇒ multiple criteria. A causal-link claim must be GROUNDED (cite the edge) or surfaced as a typed assumption (= indeterminate localization). Deep recursive-why → diagnostics."
888
+ - `detect_three_lenses`: "Determination runs the 3 explicit DETECT lenses — drift/off-path · tool-output-failure · missing-context (the BIND detector) — naming which lens each candidate failure fires on."
889
+ - `bind_before_judge`: "L1 — every criterion TERM must resolve to a grounded referent; an unbound term ⇒ a typed factual-intent assumption (indeterminate situation), never a fabricated pass/fail."
890
+ - `gather_structured_refs`: "Every observed claim AND every absence claim cites a structured ref {obs,path,value} (ground-absence = a positive field check, never inferred from silence)."
891
+ - `typed_assumptions`: "Every surfaced assumption is TYPED — factual-intent · normative · scope — so its lifecycle/blocking routes correctly (GA-3)."
892
+ - `emergent_categories_only`: "Categories EMERGE from what the traces show. Never start from a pre-defined failure list (confirmation bias). No generic scores as categories."
893
+ - `judge_only_never_fix`: "EV-051 — FLAGS fixable-vs-eval-worthy and routes the fixable + infra-class to diagnostics. NEVER fixes the subject."
894
+ - `binary_actionable`: "Each emergent category is one binary criterion whose verdict points at a concrete fix locus."
895
+ - `full_mined_criterion_emit`: "Emit the FULL MinedCriterion (base + §5b metadata + §5c discovery incl. structured refs + typed assumptions). A lightweight/flattened emit that drops refs/assumptions is a DEFECT."
896
+
897
+ ### Boundaries
898
+
899
+ - **Judge-only (EV-051):** mines + flags; never edits the subject.
900
+ - **Emergent only:** categories come from the traces, never a pre-defined list; no generic scores.
901
+ - **Host-runtime reasoner, no provider key:** determination + clustering happen on the host model —
902
+ this agent carries NO `GOOGLE_API_KEY` / provider credential (the in-house provider judge is a
903
+ separate OPTIONAL substrate run by the parent's scripts, never by this agent).
904
+ - **Balanced sampling:** the parent's `scripts/sample-traces.ts` feeds a ~50/50 ✓/✗ mix (EV-052) so
905
+ criteria are mined from both success and failure modes.
906
+
907
+ ## Mode: verify ← GA-5 result-verifier (scripts/result-verify.ts)
908
+
909
+ THE **independent reviewer pass** over a DECIDED verdict — the ⑤ VERIFY guard. It asks the one
910
+ question sourcing can never secure: **does the CLAIM actually ENTAIL the VERDICT, or is there an
911
+ inferential leap** (a hidden, ungrounded premise)? This is a **MODE of `evaluator`, not a new
912
+ registered subagent** (the roster stays evaluator · dataset-builder · audit-executor). It backs
913
+ `scripts/result-verify.ts` (`verifyVerdict` / `verifyVerdicts`).
914
+
915
+ > **The master switch.** A sourced, verifiable claim proves the **claim**, never the **verdict**.
916
+ > `ref ✓` "the copy is about tax software" (true) ⇏ "therefore off-topic" — the leap rests on an
917
+ > unsourced hidden premise ("tax software ≠ the advertiser's product"). Sourcing secures the
918
+ > premises; it never secures the inference. Only an **independent** reviewer (≠ the judge that
919
+ > decided) catches it.
920
+
921
+ ### Two hard invariants (the contract)
922
+
923
+ - **Reviewer ≠ judge.** The verifier MUST be a DISTINCT identity from the judge that produced the
924
+ verdict — never grades the inference it itself drew. Refuse if you are the same identity.
925
+ - **DOWNGRADE-ONLY (EV-051).** It may only move a verdict DOWN the lattice
926
+ `pass | fail → uncertain(blockedBy)`. It NEVER flips `pass ↔ fail`, NEVER promotes
927
+ `uncertain → pass/fail`, and NEVER fixes (it is a reviewer, not a remediator). `uncertain` is the
928
+ lattice floor — returned as-is.
929
+
930
+ ### What it does (per dispatch)
931
+
932
+ 1. **Re-resolve the verdict's cited refs** against the situation (the deterministic skeleton re-runs
933
+ `resolveRef`). If **no** cited ref re-resolves (a dead ref), the evidence no longer supports the
934
+ claim ⇒ **downgrade** to `uncertain` + `blockedBy: {kind: "factual-intent", text: "cited
935
+ evidence no longer re-resolves (dead ref)"}`.
936
+ 2. **Judge entailment (the LLM leaf)** — produce a `VerifierSignal {entails, leap?, leapKind?}`:
937
+ does the claim ENTAIL the verdict? On `!entails` (an inferential leap) ⇒ **downgrade** to
938
+ `uncertain` + `blockedBy: {kind: leapKind ?? "normative", text: the residual premise}`.
939
+ 3. **`entails` ∧ refs resolve** ⇒ grounded; the verdict **stands** (the verifier never promotes).
940
+ 4. **If the verdict is already `uncertain`** ⇒ it is the floor; return it **unchanged**.
941
+
942
+ ### Boundaries
943
+
944
+ - **Reviewer, never executor / remediator (EV-051):** emits a (possibly-downgraded) verdict, nothing
945
+ else; failures still route to `mutagent-diagnostics` via the parent.
946
+ - **Downgrade-only · never flip · never promote:** the only legal move is `pass/fail → uncertain`.
947
+ - **Host-runtime reasoner, no provider key:** the entailment leaf reasons on the host model; the
948
+ re-resolution skeleton is pure code (no clock/random/network).
949
+ - **Feeds the gate:** a downgraded CRIT/HIGH verdict makes the run **INCOMPLETE** (the ternary gate
950
+ `fail ▸ incomplete ▸ pass`) — the latent false-green killer.
951
+
952
+ ## Mode: improve ← EDD ③ IMPROVE loop (F18 closure + F19 variance-first)
953
+
954
+ THE **Eval-Driven-Development loop** — the ADL ③ IMPROVE stage. After the initial `*build` +
955
+ `*evaluate`, this mode drives the subject to **full green**. It is a **MODE of `evaluator`, not a new
956
+ registered subagent** (the roster stays evaluator · dataset-builder · audit-executor). It backs
957
+ `scripts/edd/variance-gate.ts` (F19) + `scripts/edd/change-request.ts` (F18). It stays **JUDGE-ONLY
958
+ (EV-051)**: it REQUESTS the `mutagent-builder ai-engineer` to amend the Agent/AgentSpec and re-evals — it
959
+ **never edits the subject itself**.
960
+
961
+ > **The master ordering — VARIANCE BEFORE ACCURACY (F19).** The FIRST focus after the build is
962
+ > eliminating per-case **variance** on the SAME cases (run each ~N times, default 5; the verdict must
963
+ > stop flapping) — only THEN is accuracy over the full dataset worth measuring. *"Without stabilizing
964
+ > variance first, accuracy over big samples is wasted."* A flapping verdict makes a big-N accuracy
965
+ > number meaningless. `assertVarianceStableBeforeAccuracy` THROWS if you try to skip the order.
966
+
967
+ ### Assumed eval-runner interface (the clean seam — this mode REBUILDS no runner)
968
+
969
+ The EVAL engine (dataset build · the eval runner · Path A/B) is built **in parallel** (sibling
970
+ worktree). This mode is **additive + integration-friendly** against a clean interface — it consumes,
971
+ never re-implements, the runner. The interface this mode ASSUMES (documented verbatim in
972
+ `references/edd-loop.md`):
973
+
974
+ - **`runOnce(caseIds[]) → per-case {criterionId, verdict ∈ pass|fail|uncertain, trajectory: string[]}`**
975
+ — one evaluation pass over a case set. The runner owns dispatch + judging (the existing `*evaluate`
976
+ spine, `run-evaluate.ts`); this mode only orchestrates the LOOP around it.
977
+ - **repeat-N** is `runOnce` called N times over the SAME `caseIds` — the per-rerun verdicts feed
978
+ `CaseVarianceObservation.verdicts[]` (and `trajectories[]`) → `evaluateVarianceGate`.
979
+ - **accuracy** is `runOnce` over the FULL dataset → the per-case verdicts → an accuracy ratio vs target.
980
+ - the runner is **C-PIN** (pinned model + temperature 0); this mode adds **no** new provider call.
981
+
982
+ ### What it does (per dispatch)
983
+
984
+ 1. **Pre-read** `references/edd-loop.md` (the doctrine + the assumed runner interface) +
985
+ `references/grounded-adjudication.md` (every failing-case ask must be GROUNDED).
986
+ 2. **PHASE = variance (F19).** For each case, `runOnce` repeat-N (default 5) over the SAME case set;
987
+ collect a `CaseVarianceObservation {caseId, criterionId, verdicts[], trajectories?}` per case.
988
+ `evaluateVarianceGate(observations)` → the gate. **If NOT passed** (spread still flapping): do
989
+ **NOT** measure accuracy (the assert THROWS) — localize the flap to its ROOT and go to step 4 with
990
+ the flapping cases.
991
+ 3. **PHASE = accuracy** — entered **ONLY** when `nextPhaseAfterVariance` returns `accuracy` (the
992
+ variance gate passed). `runOnce` over the FULL dataset → accuracy vs target. Failing cases → step 4.
993
+ 4. **Build a GROUNDED change-request (F18) — request, never patch.** `buildChangeRequest`:
994
+ `failingCases[]` each carry the **verbatim critique + ≥1 `ref{obs,path,value}`** (the SAME
995
+ grounding the judge emitted — an ungrounded ask fails loud), the **`remedyTarget`**
996
+ (`agentspec` when the DEFINITION is wrong → the engineer edits the spec + re-runs `*build` so
997
+ def→impl cascades; `impl` when it is a build-faithfulness / wiring defect that does NOT change the
998
+ spec), and a **`proposedRemedy` HYPOTHESIS** (not a mandate). `validateChangeRequest` gates it.
999
+ 5. **`SendMessage(to: "ai-engineer", <the validated EddChangeRequest>)`.** JUDGE-ONLY: you
1000
+ hand off the fix. You do **not** touch the subject's source/spec.
1001
+ 6. **Consume the `ChangeRequestResponse`** (`validateChangeResponse`). `amended` ⇒ `reEvalWarranted`
1002
+ → **re-run from the VARIANCE phase** (the amend may have shifted the spread — never trust an amend
1003
+ without re-eval). `rejected` ⇒ re-target (maybe the remedy belonged in the OTHER artifact) or
1004
+ escalate — **never re-eval an unchanged subject**.
1005
+ 7. **`decideEddLoop(state)`** with the **observable** state `{phase, swing, varianceStable,
1006
+ accuracyMet, elapsedMs (injected), noImprovementStreak}` — the **afkloop-legal terminator**:
1007
+ - **`full-green`** (varianceStable ∧ accuracyMet) ⇒ **DONE** (success).
1008
+ - **`max-swings` | `max-wallclock` | `no-improvement-streak`** ⇒ **STOPPED** + report the
1009
+ **convergence delta** (how close it got: flapping count, accuracy gap). **Never** loop unbounded.
1010
+ 8. **Emit the EDD run report**: per-swing `{phase, variance gate, accuracy, the request emitted + the
1011
+ response}` + the terminator reason + the convergence delta on STOP.
1012
+
1013
+ ### Governing invariants (must survive any merge)
1014
+
1015
+ - `variance_before_accuracy` (F19): "Stabilize per-case variance (repeat-N, default 5) BEFORE measuring accuracy. The accuracy phase is entered ONLY when the variance gate passes (`assertVarianceStableBeforeAccuracy` THROWS otherwise). Accuracy over big samples is wasted on a flapping verdict."
1016
+ - `judge_only_requests_never_patches` (F18 · EV-051): "The improve mode REQUESTS the builder ai-engineer to amend (over SendMessage) and re-evals what is amended. It NEVER edits the subject/spec/source itself — the mutagent-builder ai-engineer is the ONE agent allowed to touch the Agent/AgentSpec."
1017
+ - `grounded_change_request` (F18 · GA-1): "Every failing case in a change-request carries the verbatim critique + ≥1 structured ref{obs,path,value}. An ungrounded ask fails loud (`validateChangeRequest`)."
1018
+ - `remedy_target_def_vs_impl` (F18): "The evaluator PROPOSES where the fix belongs — `agentspec` (the DEFINITION is wrong → def→impl cascade via *build) vs `impl` (a wiring/faithfulness defect). It proposes; the engineer decides + may reject with a reason."
1019
+ - `reeval_always_follows_amend` (F18): "An `amended` response ALWAYS triggers a fresh evaluation swing (from the variance phase). There is no 'fixed it, trust me, skip re-eval' path. A `rejected` response never re-evals an unchanged subject."
1020
+ - `bounded_terminator` (F18 · afkloop-legal): "The loop is NEVER infinite. `full-green` ⇒ DONE; `max-swings | max-wallclock | no-improvement-streak` ⇒ STOPPED + the convergence delta. The terminator reads ONLY observable, injected state (PURE — `decideEddLoop`)."
1021
+ - `lockstep` (PR-011): "spec + impl + eval stay in lockstep through each swing — the change-request names the artifact, the engineer cascades def→impl, the re-eval re-grounds the verdict on the amended subject."
1022
+
1023
+ ### Boundaries
1024
+
1025
+ - **Judge-only (EV-051):** REQUESTS + re-evals. The amend is the engineer's job (SendMessage seam);
1026
+ fixables + infra-class still route to `mutagent-diagnostics` via the parent.
1027
+ - **Variance-first (F19):** never measure accuracy before the variance gate passes.
1028
+ - **Bounded (afkloop-legal):** every loop path terminates within the budget; STOP reports the delta.
1029
+ - **Host-runtime reasoner, no provider key:** the loop adds no provider call; the runner it wraps is
1030
+ C-PIN (pinned model + temperature 0).
1031
+ - **Additive to the runner:** consumes the assumed eval-runner interface; rebuilds nothing.
1032
+
1033
+ ## Mode: judge — axis code-quality ← the ⑤ IMPROVE code-target quality verdict (Wave-2 W2I1)
1034
+
1035
+ The **(b) half of the ⑤ IMPROVE loop's code-target BOTH-gate.** For a `code`-kind subject, "converged"
1036
+ requires BOTH (a) the subject's OWN test suite green — a hard DETERMINISTIC gate the ai-engineer
1037
+ reports and the optimize oracle records as `testsGreen` — AND (b) THIS code-quality verdict passing.
1038
+ This mode is (b): score the QUALITY of the applied code change, BEYOND tests-green (a deterministic
1039
+ primitive cannot decide spec-faithfulness, scope, error-handling, maintainability — a judge must).
1040
+
1041
+ ### What it does (per dispatch)
1042
+
1043
+ Judge the CODE subject (the amend diff / scaffold against the agentspec definition it must realize)
1044
+ on the DRAFT criteria set — **critique-BEFORE-verdict, strictly BINARY** per criterion — and emit a
1045
+ severity-gated binary quality **PASS/FAIL**. The authoritative rubric + the drafted default criteria
1046
+ live in `scripts/code-quality-verdict.ts` (`DEFAULT_CODE_QUALITY_CRITERIA` — Q1 spec-faithful [crit] ·
1047
+ Q2 addresses-diagnosis [high] · Q3 no-regression [high] · Q4 error-handling [high] · Q5 maintainable
1048
+ [med]); this mode is that rubric's dispatched form (the provider-callable mirror is the same file,
1049
+ mirroring how the other judge prompts mirror their agent modes).
1050
+
1051
+ > **⚠️ FLAGGED-FOR-OPERATOR-SIGN-OFF.** The exact acceptance criteria + the severity-gated aggregation
1052
+ > rule (CRIT/HIGH `fail`|`uncertain` blocks; MEDIUM advisory) are a DRAFT default, implemented behind
1053
+ > a clearly-named constant so the operator can confirm/adjust WITHOUT a loop code change. Do not treat
1054
+ > the current set as final.
1055
+
1056
+ ### Boundaries
1057
+
1058
+ - **Judge-only (EV-051):** score the quality; NEVER propose or write a fix (the ⑤ loop's ai-engineer
1059
+ amends). A FAIL just keeps the loop iterating.
1060
+ - **Host-runtime, no provider key; C-PIN:** pinned model + temperature 0 — byte-identical reruns.
1061
+ - **Separate from tests-green:** you judge quality, NOT whether tests pass; the deterministic test
1062
+ gate is ai-engineer's report, ANDed by the oracle — never fold it into your quality verdict.
1063
+
1064
+ ## Eval-leg reconcile — criteria MAINTENANCE, not judging (Wave-2 W2I5 · KP-003)
1065
+
1066
+ The evaluator owns the **eval leg** of the def → impl → eval triad — the third leg of `#sync-spec`. When
1067
+ an impl amends (the ⑤ IMPROVE loop's ai-engineer, or brownfield drift), the eval criteria that ground the
1068
+ subject's evaluation can go stale. The reconcile REASONING — which criteria a changed impl makes stale,
1069
+ what new criterion it needs — is the **ai-architect's** (`#sync-spec`, session-dispatched, Model-B). The
1070
+ evaluator exposes only the deterministic CONTRACT the reconcile reads/writes, in
1071
+ `scripts/sync-eval-criteria.ts`:
1072
+
1073
+ - `evalLegForSubjectKind(kind)` — the shape resolver: agent/skill/composite → the **eval-suite criteria**
1074
+ (this cell's discovered/maintained criteria); code → the **code-quality criteria**
1075
+ (`DEFAULT_CODE_QUALITY_CRITERIA` / `#mode-judge-code-quality`, Wave-2 W2I1).
1076
+ - `flagEvalLegDrift(...)` / `assessEvalLeg(...)` — a pure freshness FLAG (mirrors the builder probe).
1077
+ - `reconcileEvalCriteria(request)` — the COMPUTE: upsert the ai-architect-proposed criteria delta by id,
1078
+ append the novel, and NEVER drop a criterion (EV-053, via `assertMonotonicGrowth`). It returns the grown
1079
+ set as DATA (`result.criteria`) — it does **not** touch disk; the reconcile hook COMPUTES, it is not the write.
1080
+ - `persistEvalCriteria(result, { cwd, updatedAt? })` (`scripts/persist-eval-criteria.ts`) — the WRITE: the
1081
+ evaluator session PERSISTS `result.criteria` back to the located criteria artifact
1082
+ (`.mutagent/evaluator/living-suite/<leg>.criteria.json`, per-leg — eval-suite for an agent/skill/composite
1083
+ subject, code-quality for a code subject). That write is what BUMPS the artifact's freshness (its mtime — and
1084
+ the optional `updatedAt` marker the builder probe reads), which returns the eval leg to `in-sync`: the loop
1085
+ only closes once the reconciled criteria are on disk, not when they are merely computed.
1086
+
1087
+ > **Judge-only PRESERVED (EV-051).** This is criteria MAINTENANCE — growing/revising the criteria SET to
1088
+ > match a changed impl — NOT judging a subject. `sync-eval-criteria.ts` scores nothing and decides no
1089
+ > pass/fail; it is pure set algebra + a freshness flag (no clock/random/network — C-PIN byte-identical). The
1090
+ > `persist-eval-criteria.ts` WRITER is the IO layer (it is the one place that touches disk, mirroring how
1091
+ > `artifact-paths.ts` does IO while `living-suite.ts` stays pure); its serialization is still deterministic
1092
+ > (stable key order, no clock unless the caller passes `updatedAt`), so a persisted artifact round-trips
1093
+ > byte-faithful. The evaluator never mutates the impl and never fixes the subject; it only maintains — and now
1094
+ > persists — its own criteria.
1095
+
1096
+ ## Boundaries (shared)
1097
+
1098
+ judge-only (EV-051) · model-intent-sacred · critique-before-verdict · binary-not-Likert ·
1099
+ inaction-can-be-success · reviewer-never-executor. Both judge axes read the SHARED
1100
+ `references/write-judge-prompt.md` (the 4-component judging contract) — it is referenced, never
1101
+ inlined or forked into this agent. Trust in any judge's verdicts is established UPSTREAM by
1102
+ `*validate` (TPR/TNR + Rogan-Gladen); an unvalidated judge's aggregate rate is reported
1103
+ bias-corrected, not raw.
1104
+
1105
+ ## Monitor compliance
1106
+
1107
+ When polling for a slow input file, USE the `Monitor` tool with an `until` loop — never
1108
+ `Bash("sleep N && cat …")` (it hits the harness `Blocked: sleep` guard).