session-orchestrator 5.2.0 → 5.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/architecture/SKILL.md +3 -1
- package/.agents/skills/autopilot/SKILL.md +5 -1
- package/.agents/skills/autopilot/agents/openai.yaml +5 -0
- package/.agents/skills/bootstrap/SKILL.md +5 -1
- package/.agents/skills/bootstrap/agents/openai.yaml +5 -0
- package/.agents/skills/brainstorm/SKILL.md +5 -1
- package/.agents/skills/brainstorm/agents/openai.yaml +5 -0
- package/.agents/skills/claude-md-drift-check/SKILL.md +4 -2
- package/.agents/skills/close/SKILL.md +5 -1
- package/.agents/skills/close/agents/openai.yaml +5 -0
- package/.agents/skills/convergence-monitoring/SKILL.md +4 -2
- package/.agents/skills/debug/SKILL.md +5 -1
- package/.agents/skills/debug/agents/openai.yaml +5 -0
- package/.agents/skills/discovery/SKILL.md +5 -1
- package/.agents/skills/discovery/agents/openai.yaml +5 -0
- package/.agents/skills/dispatcher/SKILL.md +5 -1
- package/.agents/skills/dispatcher/agents/openai.yaml +5 -0
- package/.agents/skills/docs-orchestrator/SKILL.md +3 -1
- package/.agents/skills/ecosystem-health/SKILL.md +3 -1
- package/.agents/skills/eli5/SKILL.md +5 -1
- package/.agents/skills/eli5/agents/openai.yaml +5 -0
- package/.agents/skills/eval/SKILL.md +6 -2
- package/.agents/skills/eval/agents/openai.yaml +5 -0
- package/.agents/skills/evolve/SKILL.md +6 -2
- package/.agents/skills/evolve/agents/openai.yaml +5 -0
- package/.agents/skills/frontmatter-guard/SKILL.md +3 -1
- package/.agents/skills/gitlab-ops/SKILL.md +3 -1
- package/.agents/skills/gitlab-portfolio/SKILL.md +3 -1
- package/.agents/skills/go/SKILL.md +5 -1
- package/.agents/skills/go/agents/openai.yaml +5 -0
- package/.agents/skills/grill/SKILL.md +5 -1
- package/.agents/skills/grill/agents/openai.yaml +5 -0
- package/.agents/skills/harness-audit/SKILL.md +5 -1
- package/.agents/skills/harness-audit/agents/openai.yaml +5 -0
- package/.agents/skills/hook-development/SKILL.md +3 -1
- package/.agents/skills/mcp-builder/SKILL.md +3 -1
- package/.agents/skills/memory-cleanup/SKILL.md +5 -1
- package/.agents/skills/memory-cleanup/agents/openai.yaml +5 -0
- package/.agents/skills/mode-selector/SKILL.md +3 -1
- package/.agents/skills/navigator/SKILL.md +22 -0
- package/.agents/skills/navigator/agents/openai.yaml +5 -0
- package/.agents/skills/npm-publish/SKILL.md +4 -2
- package/.agents/skills/peekaboo-driver/SKILL.md +3 -1
- package/.agents/skills/persona-panel/SKILL.md +5 -1
- package/.agents/skills/persona-panel/agents/openai.yaml +5 -0
- package/.agents/skills/plan/SKILL.md +5 -1
- package/.agents/skills/plan/agents/openai.yaml +5 -0
- package/.agents/skills/playwright-driver/SKILL.md +3 -1
- package/.agents/skills/portfolio/SKILL.md +5 -1
- package/.agents/skills/portfolio/agents/openai.yaml +5 -0
- package/.agents/skills/quality-gates/SKILL.md +3 -1
- package/.agents/skills/reconcile/SKILL.md +5 -1
- package/.agents/skills/reconcile/agents/openai.yaml +5 -0
- package/.agents/skills/release/SKILL.md +5 -1
- package/.agents/skills/release/agents/openai.yaml +5 -0
- package/.agents/skills/remote-offload/SKILL.md +3 -1
- package/.agents/skills/repo-audit/SKILL.md +5 -1
- package/.agents/skills/repo-audit/agents/openai.yaml +5 -0
- package/.agents/skills/session/SKILL.md +21 -0
- package/.agents/skills/session/agents/openai.yaml +5 -0
- package/.agents/skills/session-end/SKILL.md +3 -1
- package/.agents/skills/session-plan/SKILL.md +3 -1
- package/.agents/skills/session-start/SKILL.md +3 -1
- package/.agents/skills/spinout/SKILL.md +5 -1
- package/.agents/skills/spinout/agents/openai.yaml +5 -0
- package/.agents/skills/sunset-review/SKILL.md +5 -1
- package/.agents/skills/sunset-review/agents/openai.yaml +5 -0
- package/.agents/skills/templates-ack/SKILL.md +21 -0
- package/.agents/skills/templates-ack/agents/openai.yaml +5 -0
- package/.agents/skills/test/SKILL.md +5 -1
- package/.agents/skills/test/agents/openai.yaml +5 -0
- package/.agents/skills/test-audit/SKILL.md +21 -0
- package/.agents/skills/test-audit/agents/openai.yaml +5 -0
- package/.agents/skills/test-runner/SKILL.md +3 -1
- package/.agents/skills/tmux-layout/SKILL.md +3 -1
- package/.agents/skills/using-orchestrator/SKILL.md +3 -1
- package/.agents/skills/ux-grill/SKILL.md +5 -1
- package/.agents/skills/ux-grill/agents/openai.yaml +5 -0
- package/.agents/skills/vault-mirror/SKILL.md +3 -1
- package/.agents/skills/vault-sync/SKILL.md +3 -1
- package/.agents/skills/wave-executor/SKILL.md +3 -1
- package/.agents/skills/write-executable-plan/SKILL.md +3 -1
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.codex-plugin/plugin.json +4 -4
- package/.codex-plugin/skills/claude-md-drift-check/SKILL.md +1 -1
- package/.codex-plugin/skills/convergence-monitoring/SKILL.md +1 -3
- package/.codex-plugin/skills/eval/SKILL.md +1 -1
- package/.codex-plugin/skills/evolve/SKILL.md +1 -1
- package/.codex-plugin/skills/navigator/SKILL.md +22 -0
- package/.codex-plugin/skills/{convergence-monitoring → navigator}/agents/openai.yaml +2 -2
- package/.codex-plugin/skills/npm-publish/SKILL.md +1 -3
- package/.codex-plugin/skills/session/SKILL.md +1 -1
- package/.codex-plugin/skills/test-audit/SKILL.md +21 -0
- package/.codex-plugin/skills/{npm-publish → test-audit}/agents/openai.yaml +1 -1
- package/.cursor/commands/eval.md +1 -1
- package/.cursor/commands/navigator.md +15 -0
- package/.cursor/commands/session.md +1 -1
- package/.cursor/commands/test-audit.md +14 -0
- package/.cursor/rules/000-session-orchestrator.mdc +0 -2
- package/.cursor/rules/050-plan.mdc +1 -1
- package/.cursor/rules/060-evolve.mdc +27 -12
- package/.cursor/skills/claude-md-drift-check/SKILL.md +1 -1
- package/.cursor/skills/convergence-monitoring/SKILL.md +1 -0
- package/.cursor/skills/eval/SKILL.md +1 -1
- package/.cursor/skills/navigator/SKILL.md +13 -0
- package/.cursor/skills/npm-publish/SKILL.md +1 -0
- package/.cursor/skills/test-audit/SKILL.md +12 -0
- package/.cursor-plugin/plugin.json +1 -1
- package/.orchestrator/policy/blocked-commands.json +12 -3
- package/AGENTS.md +9 -30
- package/CHANGELOG.md +206 -0
- package/README.md +13 -9
- package/SECURITY.md +12 -0
- package/agents/dialectic-deriver.md +13 -10
- package/agents/eval-judge.md +67 -45
- package/agents/skill-applied-judge.md +34 -19
- package/commands/session.md +7 -3
- package/docs/agent-authoring.md +2 -2
- package/docs/baseline.md +22 -7
- package/docs/ci-setup.md +57 -4
- package/docs/codex-setup.md +14 -2
- package/docs/components.md +12 -10
- package/docs/events-schema.md +65 -15
- package/docs/persona-panel.md +5 -4
- package/docs/rule-authoring.md +84 -17
- package/docs/scope-collision-guard.md +26 -4
- package/docs/session-config-reference.md +154 -12
- package/docs/session-config-template.md +41 -3
- package/docs/telemetry.md +2 -0
- package/hooks/_lib/hook-import-set.json +87 -6
- package/hooks/_lib/subagent-transcript.mjs +582 -31
- package/hooks/config-protection.mjs +11 -3
- package/hooks/cwd-change-restore.mjs +11 -3
- package/hooks/enforce-commands.mjs +70 -23
- package/hooks/enforce-scope.mjs +143 -33
- package/hooks/hooks-codex.json +1 -1
- package/hooks/hooks-cursor.json +5 -0
- package/hooks/hooks-pi.json +5 -0
- package/hooks/hooks.json +6 -1
- package/hooks/loop-guard.mjs +11 -3
- package/hooks/on-session-end.mjs +58 -23
- package/hooks/on-session-start.mjs +101 -11
- package/hooks/on-stop.mjs +170 -22
- package/hooks/operator-steer.mjs +11 -3
- package/hooks/post-bash-issue-budget-refund.mjs +18 -8
- package/hooks/post-bash-write-verify.mjs +3 -2
- package/hooks/post-edit-import-probe.mjs +17 -9
- package/hooks/post-edit-validate.mjs +13 -5
- package/hooks/post-subagent-discovery-validator.mjs +98 -13
- package/hooks/post-tool-batch-wave-signal.mjs +202 -38
- package/hooks/post-tool-failure-corrective-context.mjs +11 -5
- package/hooks/post-tooluse-frontend-slop.mjs +10 -4
- package/hooks/pre-auq-clarity.mjs +15 -2
- package/hooks/pre-bash-destructive-guard.mjs +80 -9
- package/hooks/pre-bash-issue-budget.mjs +16 -11
- package/hooks/pre-bash-memory-propose-audit.mjs +137 -56
- package/hooks/pre-bash-sessions-ledger-guard.mjs +419 -22
- package/hooks/pre-bash-staging-fence.mjs +335 -31
- package/hooks/pre-bash-templates-first.mjs +19 -14
- package/hooks/pre-edit-sessions-ledger-guard.mjs +162 -0
- package/hooks/pre-task-scope-disjoint.mjs +792 -74
- package/hooks/subagent-telemetry.mjs +111 -38
- package/hooks/wave-scope-commit-guard.mjs +197 -100
- package/monitors/monitors.json +1 -1
- package/output-styles/wave-summary.md +1 -1
- package/package.json +3 -3
- package/pi/prompts/eval.md +1 -1
- package/pi/prompts/navigator.md +12 -0
- package/pi/prompts/session.md +1 -1
- package/pi/prompts/test-audit.md +12 -0
- package/rules/README.md +7 -2
- package/rules/_index.md +3 -3
- package/rules/always-on/bash-harness-pitfalls.md +7 -1
- package/rules/opt-in-domain/prompt-caching.md +6 -2
- package/rules/opt-in-stack/backend-data.md +10 -2
- package/rules/opt-in-stack/backend.md +10 -4
- package/rules/opt-in-stack/frontend.md +7 -2
- package/rules/opt-in-stack/security-web.md +10 -4
- package/rules/opt-in-stack/swift.md +6 -2
- package/scripts/apply-session-learnings.mjs +492 -0
- package/scripts/archive-closed-prds.mjs +2 -1
- package/scripts/autopilot.mjs +70 -53
- package/scripts/backfill-abandoned-sessions.mjs +118 -16
- package/scripts/baseline-archetypes.mjs +1 -1
- package/scripts/check-package-manager.mjs +62 -5
- package/scripts/check-sessions-integrity.mjs +300 -0
- package/scripts/cursor-install.mjs +1 -4
- package/scripts/dialectic-deriver.mjs +50 -13
- package/scripts/emit-session.mjs +371 -31
- package/scripts/eval-session.mjs +65 -3
- package/scripts/export-hw-learnings.mjs +22 -3
- package/scripts/fleet-checkin.mjs +147 -0
- package/scripts/generate-agents-skills.mjs +102 -29
- package/scripts/generate-cursor-adapter.mjs +61 -16
- package/scripts/lib/agent-frontmatter.mjs +3 -3
- package/scripts/lib/agent-status.mjs +2 -31
- package/scripts/lib/auq/clarity.mjs +10 -2
- package/scripts/lib/auq/parse.mjs +12 -31
- package/scripts/lib/auq/schema.mjs +56 -41
- package/scripts/lib/auto-dialectic.mjs +304 -15
- package/scripts/lib/auto-dream.mjs +7 -6
- package/scripts/lib/autopilot/flags.mjs +18 -2
- package/scripts/lib/autopilot/kill-switches.mjs +6 -3
- package/scripts/lib/autopilot/loop.mjs +19 -2
- package/scripts/lib/autopilot/stall-sampler.mjs +80 -23
- package/scripts/lib/baseline-archetypes.mjs +41 -22
- package/scripts/lib/ci-status-banner.mjs +376 -16
- package/scripts/lib/command-blocker.mjs +275 -28
- package/scripts/lib/config/dialectic.mjs +12 -3
- package/scripts/lib/config/gate.mjs +74 -0
- package/scripts/lib/config/host-paths.mjs +79 -0
- package/scripts/lib/config/persona-gate-wave.mjs +3 -3
- package/scripts/lib/config/reaper.mjs +163 -0
- package/scripts/lib/config-schema.mjs +13 -0
- package/scripts/lib/config.mjs +85 -10
- package/scripts/lib/convergence-monitor.mjs +74 -11
- package/scripts/lib/ecosystem-health.mjs +11 -0
- package/scripts/lib/eval/engine.mjs +421 -53
- package/scripts/lib/eval/judge.mjs +463 -40
- package/scripts/lib/eval/schema.mjs +10 -1
- package/scripts/lib/events-rotation.mjs +221 -25
- package/scripts/lib/events-schema.mjs +114 -0
- package/scripts/lib/events.mjs +545 -7
- package/scripts/lib/fleet-protocol.mjs +270 -0
- package/scripts/lib/frontmatter-guard.mjs +21 -10
- package/scripts/lib/gates/gate-baseline.mjs +40 -5
- package/scripts/lib/gates/gate-full.mjs +35 -5
- package/scripts/lib/gates/gate-helpers.mjs +336 -23
- package/scripts/lib/gates/gate-incremental.mjs +49 -23
- package/scripts/lib/gates/gate-per-file.mjs +42 -8
- package/scripts/lib/gitlab-portfolio/markdown-writer.mjs +6 -1
- package/scripts/lib/instruction-budget-guard.mjs +208 -8
- package/scripts/lib/io.mjs +42 -8
- package/scripts/lib/issue-close-strip-labels.mjs +207 -49
- package/scripts/lib/js-mask.mjs +197 -0
- package/scripts/lib/learnings/evolve-telemetry.mjs +11 -7
- package/scripts/lib/learnings/expiry-sweep.mjs +34 -3
- package/scripts/lib/learnings/io.mjs +154 -26
- package/scripts/lib/learnings/schema.mjs +163 -15
- package/scripts/lib/learnings/sizing-subject.mjs +2 -2
- package/scripts/lib/maintenance-due-banner.mjs +53 -88
- package/scripts/lib/memory-proposals/schema.mjs +11 -5
- package/scripts/lib/named-baseline-resolver.mjs +53 -28
- package/scripts/lib/named-vault-resolver.mjs +104 -5
- package/scripts/lib/orphan-reaper/defaults.mjs +41 -0
- package/scripts/lib/orphan-reaper/ps-snapshot.mjs +181 -0
- package/scripts/lib/orphan-reaper/reaper-audit.mjs +291 -0
- package/scripts/lib/orphan-reaper/reaper-decide.mjs +395 -0
- package/scripts/lib/orphan-reaper/scan-throttle.mjs +92 -0
- package/scripts/lib/orphan-reaper/scan.mjs +506 -0
- package/scripts/lib/orphan-reaper.mjs +316 -0
- package/scripts/lib/owner-yaml.mjs +104 -6
- package/scripts/lib/peer-cards/merger.mjs +48 -10
- package/scripts/lib/peer-cards/reader.mjs +78 -2
- package/scripts/lib/persona-panel/catalog-loader.mjs +1 -1
- package/scripts/lib/process-group.mjs +950 -0
- package/scripts/lib/project-hygiene.mjs +21 -1
- package/scripts/lib/quality-gate.mjs +152 -33
- package/scripts/lib/reconcile/backlog.mjs +368 -0
- package/scripts/lib/reconcile/engine.mjs +55 -188
- package/scripts/lib/reconcile/rule-expiry-sweep.mjs +302 -60
- package/scripts/lib/reconcile/sanitize.mjs +69 -3
- package/scripts/lib/reconcile-nudge-banner.mjs +138 -45
- package/scripts/lib/resource-probe/parsers.mjs +31 -0
- package/scripts/lib/resource-probe.mjs +1 -1
- package/scripts/lib/rule-loader.mjs +141 -74
- package/scripts/lib/rules-sync.mjs +27 -12
- package/scripts/lib/scope-baseline.mjs +3 -4
- package/scripts/lib/scope-echo.mjs +39 -2
- package/scripts/lib/scope-gate.mjs +605 -1
- package/scripts/lib/session-close-backfill.mjs +33 -6
- package/scripts/lib/session-id.mjs +9 -20
- package/scripts/lib/session-invocation.mjs +20 -0
- package/scripts/lib/session-record-repair.mjs +183 -26
- package/scripts/lib/session-registry.mjs +27 -2
- package/scripts/lib/session-schema/constants.mjs +56 -2
- package/scripts/lib/session-schema/normalizer.mjs +56 -4
- package/scripts/lib/session-schema/validator.mjs +28 -1
- package/scripts/lib/session-schema.mjs +8 -3
- package/scripts/lib/session-start-probes.mjs +95 -10
- package/scripts/lib/session-token-rollup.mjs +245 -52
- package/scripts/lib/sessions-canonical.mjs +118 -4
- package/scripts/lib/sessions-integrity-banner.mjs +7 -1
- package/scripts/lib/sessions-staleness-banner.mjs +193 -51
- package/scripts/lib/sh-quote.mjs +29 -0
- package/scripts/lib/skill-evidence-window.mjs +891 -0
- package/scripts/lib/skill-evolution/candidate-intake.mjs +133 -12
- package/scripts/lib/skill-evolution/engine.mjs +18 -9
- package/scripts/lib/skill-judge.mjs +45 -3
- package/scripts/lib/subagents-schema.mjs +30 -3
- package/scripts/lib/tail-window.mjs +56 -0
- package/scripts/lib/telemetry/pricing.mjs +118 -71
- package/scripts/lib/telemetry/schema.mjs +30 -0
- package/scripts/lib/telemetry/sync.mjs +61 -6
- package/scripts/lib/telemetry-flush-health-banner.mjs +4 -22
- package/scripts/lib/test-runner/issue-reconcile.mjs +48 -16
- package/scripts/lib/tests-src-ratio.mjs +140 -23
- package/scripts/lib/tmux-layout/layouts.mjs +1 -11
- package/scripts/lib/tmux-layout/telemetry-stats.mjs +72 -13
- package/scripts/lib/tmux-layout/tmux-shell.mjs +2 -11
- package/scripts/lib/user-invocable-skills.mjs +23 -3
- package/scripts/lib/ux-grill/reconcile.mjs +48 -22
- package/scripts/lib/validate/check-agents-skills.mjs +26 -15
- package/scripts/lib/validate/check-agents.mjs +1 -1
- package/scripts/lib/validate/check-cursor-adapter.mjs +1 -0
- package/scripts/lib/validate/check-entry-guard.mjs +13 -50
- package/scripts/lib/validate/check-hook-entry-guards.mjs +636 -0
- package/scripts/lib/validate/check-hooks-symmetry.mjs +5 -0
- package/scripts/lib/validate/check-learnings-shell-writes.mjs +214 -0
- package/scripts/lib/validate/check-pi-prompts.mjs +1 -0
- package/scripts/lib/validate/check-rules.mjs +7 -5
- package/scripts/lib/validate/check-skill-links.mjs +9 -1
- package/scripts/lib/validate/check-skill-script-paths.mjs +239 -27
- package/scripts/lib/validate/check-test-git-config-target.mjs +24 -34
- package/scripts/lib/validate/check-test-value-bans.mjs +87 -14
- package/scripts/lib/validate/check-unicode-safety.mjs +1 -0
- package/scripts/lib/validate/check-untracked-test-deps.mjs +7 -102
- package/scripts/lib/validate/check-unwired-features.mjs +130 -27
- package/scripts/lib/validate/check-validator-registration.mjs +34 -10
- package/scripts/lib/validate/confidential-names.mjs +10 -0
- package/scripts/lib/validate-vendored-rules.mjs +210 -84
- package/scripts/lib/vault-mirror/namespace.mjs +46 -8
- package/scripts/lib/vault-mirror/process.mjs +10 -3
- package/scripts/lib/vault-mirror/render-sessions.mjs +12 -2
- package/scripts/lib/vault-mirror/telemetry.mjs +8 -1
- package/scripts/lib/vault-status/board-writer.mjs +48 -11
- package/scripts/lib/vault-status/narrative-mirror.mjs +71 -16
- package/scripts/lib/vault-yaml.mjs +118 -0
- package/scripts/lib/wave-executor/dispatch-common.mjs +10 -18
- package/scripts/lib/wave-executor/foreign-dispatch.mjs +1 -1
- package/scripts/lib/wave-executor/remote-dispatch.mjs +1 -1
- package/scripts/lib/wave-resource-gate.mjs +34 -8
- package/scripts/lib/worktree/lifecycle.mjs +153 -1
- package/scripts/memory-propose.mjs +18 -5
- package/scripts/release-session-lock.mjs +306 -0
- package/scripts/release.mjs +40 -8
- package/scripts/repair-invalid-sessions.mjs +37 -4
- package/scripts/resolve-session-invocation.mjs +59 -0
- package/scripts/run-migrate-v2-cross-repo.mjs +158 -30
- package/scripts/run-quality-gate.mjs +357 -197
- package/scripts/site-numbers.mjs +163 -5
- package/scripts/sweep-expired-learnings.mjs +56 -5
- package/scripts/sweep-expired-rules.mjs +16 -4
- package/scripts/validate-plugin.mjs +19 -0
- package/scripts/validate-wave-scope.mjs +32 -105
- package/scripts/vault-integration-watcher.mjs +2 -1
- package/scripts/vault-mirror.mjs +81 -47
- package/scripts/wave-scope-binding.mjs +2 -1
- package/skills/_shared/config-reading.md +3 -3
- package/skills/_shared/fleet-protocol.md +131 -0
- package/skills/_shared/platform-tools.md +23 -11
- package/skills/_shared/state-ownership.md +5 -0
- package/skills/autopilot/SKILL.md +47 -18
- package/skills/bootstrap/SKILL.md +1 -1
- package/skills/bootstrap/_shared-template.md +1 -1
- package/skills/bootstrap/fast-template.md +1 -1
- package/skills/bootstrap/private-contract.md +2 -1
- package/skills/bootstrap/public-fallback.md +6 -3
- package/skills/bootstrap/references/bootstrap-retroactive-flow.md +2 -1
- package/skills/claude-md-drift-check/SKILL.md +10 -10
- package/skills/claude-md-drift-check/checker.mjs +26 -18
- package/skills/convergence-monitoring/README.md +8 -1
- package/skills/convergence-monitoring/SIGNALS.md +50 -6
- package/skills/convergence-monitoring/SKILL.md +15 -6
- package/skills/eval/SKILL.md +39 -24
- package/skills/eval/rubric-v1.md +1 -0
- package/skills/eval/rubric-v2.md +457 -0
- package/skills/evolve/SKILL.md +3 -3
- package/skills/evolve/references/evolve-analyze-mode.md +10 -6
- package/skills/evolve/references/evolve-dialectic-mode.md +42 -25
- package/skills/gitlab-ops/SKILL.md +3 -2
- package/skills/memory-cleanup/SKILL.md +1 -1
- package/skills/navigator/SKILL.md +193 -0
- package/skills/navigator/references/account-check.sh +87 -0
- package/skills/navigator/references/fleet-decisions.mjs +231 -0
- package/skills/navigator/references/slot-check.sh +114 -0
- package/skills/npm-publish/SKILL.md +1 -1
- package/skills/persona-panel/SKILL.md +2 -2
- package/skills/persona-panel/persona-format.md +2 -2
- package/skills/persona-panel/presets/designer-lens.md +1 -1
- package/skills/persona-panel/presets/engineer-lens.md +1 -1
- package/skills/persona-panel/presets/pm-lens.md +1 -1
- package/skills/plan/mode-retro.md +19 -4
- package/skills/quality-gates/SKILL.md +6 -4
- package/skills/reconcile/SKILL.md +11 -0
- package/skills/remote-offload/SKILL.md +4 -2
- package/skills/session-end/SKILL.md +13 -16
- package/skills/session-end/discovery-scan.md +1 -1
- package/skills/session-end/learning-patterns.md +22 -12
- package/skills/session-end/metrics-collection.md +5 -6
- package/skills/session-end/phase-3-6-tail.md +55 -9
- package/skills/session-end/references/phase-3-documentation-updates.md +2 -2
- package/skills/session-end/references/phase-4a-worktree-cleanup.md +2 -0
- package/skills/session-end/references/phase-5-issue-cleanup.md +9 -14
- package/skills/session-end/references/session-summary-template.md +1 -1
- package/skills/session-end/session-metrics-write.md +32 -60
- package/skills/session-plan/SKILL.md +17 -8
- package/skills/session-plan/references/session-plan-task-classification.md +2 -2
- package/skills/session-start/SKILL.md +6 -0
- package/skills/session-start/references/operations-contract.md +4 -0
- package/skills/session-start/references/phase-1-7-vault-status-board.md +1 -1
- package/skills/session-start/references/phase-4-ssot-environment-check.md +4 -3
- package/skills/test-audit/SKILL.md +152 -0
- package/skills/test-audit/references/campaign.md +111 -0
- package/skills/test-audit/references/mutate.sh +50 -0
- package/skills/ux-grill/SKILL.md +1 -1
- package/skills/wave-executor/SKILL.md +9 -5
- package/skills/wave-executor/circuit-breaker.md +2 -0
- package/skills/wave-executor/references/wave-executor-state-init.md +8 -3
- package/skills/wave-executor/references/wave-loop-dispatch.md +5 -2
- package/skills/wave-executor/references/wave-loop-review.md +3 -5
- package/templates/personas/README.md +5 -5
- package/templates/personas/accounting-compliance.v1.md +1 -1
- package/templates/personas/accounting-tax-advisor.v1.md +1 -1
- package/templates/personas/buyer-p1-cto.v1.md +1 -1
- package/templates/personas/buyer-p2-kanzlei.v1.md +1 -1
- package/templates/personas/buyer-p3-build.v1.md +1 -1
- package/templates/personas/buyer-p4-tech-veto.v1.md +1 -1
- package/templates/personas/buyer-p5-solo.v1.md +1 -1
- package/templates/personas/buyer-p6-ld.v1.md +1 -1
- package/templates/personas/klima-ai-expert.v1.md +1 -1
- package/templates/personas/klima-physicist.v1.md +1 -1
- package/.cursor/commands/convergence-monitoring.md +0 -13
- package/.cursor/commands/npm-publish.md +0 -13
- package/pi/prompts/convergence-monitoring.md +0 -11
- package/pi/prompts/npm-publish.md +0 -11
|
@@ -182,7 +182,7 @@ inherited Fast passes the confirmed contract ID and records all created paths.
|
|
|
182
182
|
|
|
183
183
|
Idempotency is handled by the writer itself:
|
|
184
184
|
- Missing → create
|
|
185
|
-
- Exists, plugin-owned (
|
|
185
|
+
- Exists, plugin-owned (the `<!-- source: session-orchestrator plugin ... -->` header is on line 1, or is the first line after a line-1 frontmatter's closing `---` — `isPluginOwnedContent()`, #1449) and byte-identical → skip silently
|
|
186
186
|
- Exists, plugin-owned and stale → overwrite (the plugin copy is canonical)
|
|
187
187
|
- Exists WITHOUT that header → preserved untouched (a repo-private rule the operator authored)
|
|
188
188
|
|
|
@@ -22,7 +22,8 @@ BOOTSTRAP_CONTRACT=$(node "$PLUGIN_ROOT/scripts/baseline-archetypes.mjs" \
|
|
|
22
22
|
export BOOTSTRAP_CONTRACT
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
`status: error` aborts before scaffolding. Its reason
|
|
25
|
+
`status: error` aborts before scaffolding. Its reason — and, for
|
|
26
|
+
`producer-missing`, its path-free `message` — is safe to report; do not
|
|
26
27
|
print baseline paths, raw producer diagnostics, or private catalog files.
|
|
27
28
|
`insufficient-evidence` is a selection state, not permission to use `node-minimal`.
|
|
28
29
|
Fast tier has no archetype; use the plugin's minimal instruction-file flow and
|
|
@@ -23,12 +23,15 @@ export BOOTSTRAP_CONTRACT PATH_TYPE
|
|
|
23
23
|
|----------------|--------------------|
|
|
24
24
|
| No configured baseline, empty value, or missing directory | `public`; use bundled templates |
|
|
25
25
|
| Existing baseline with valid reduced schema v1 export | `private`; retain `archetypes` and `selected` |
|
|
26
|
-
| Existing baseline with missing
|
|
26
|
+
| Existing baseline directory without its `scripts/archetype-manifest.mjs` exporter (`reason: producer-missing`) | Abort with the sanitized reason and its `message`; tell the user which interface is missing and offer the two ways forward: migrate the baseline to the exporter contract ([`docs/baseline.md`](../../docs/baseline.md) § How bootstrap finds it) or unset the configured baseline path (`SO_BASELINE_PATH` / matching `owner.yaml` `baselines` entry / `paths.baseline-path` / `plan-baseline-path`) to run the public path. This is the user's decision — never silently downgrade to public. <!-- path-check: example --> |
|
|
27
|
+
| Existing baseline with invalid export, unsafe source (symlink, path escape), or producer failure | Abort; report the sanitized error reason |
|
|
27
28
|
| Valid private contract but no matching repository markers | `private`, `selected: null`; require selection for Standard/Deep |
|
|
28
29
|
|
|
29
30
|
Lookup is offline and read-only. A broken configured contract never silently
|
|
30
|
-
switches to a public archetype.
|
|
31
|
-
|
|
31
|
+
switches to a public archetype. `producer-missing` is the one reason with a
|
|
32
|
+
documented migration path (migrate or unconfigure, above); even then the public
|
|
33
|
+
path runs only after the user unsets the configured baseline. Missing
|
|
34
|
+
directories retain the established public fallback. Never guess a baseline path or fetch one automatically.
|
|
32
35
|
|
|
33
36
|
## Private Path
|
|
34
37
|
|
|
@@ -28,7 +28,8 @@ Entered when `$ARGUMENTS` contains `--retroactive`. Writes the lock file and, pe
|
|
|
28
28
|
4. **Infer archetype.** Run Phase 0.5's read-only source detection. For a private
|
|
29
29
|
contract, use its detected `selected.id`; retain `null` with an explicit
|
|
30
30
|
`insufficient-evidence` report if no markers match. An invalid configured
|
|
31
|
-
contract aborts
|
|
31
|
+
contract aborts (for `producer-missing`, report its path-free `message` with
|
|
32
|
+
the reason). Do not scaffold or apply rules in this retroactive flow.
|
|
32
33
|
For the public path, use best-effort detection from existing files:
|
|
33
34
|
- `pyproject.toml` present → `python-uv`
|
|
34
35
|
- `package.json` with `next` in dependencies → `nextjs-minimal`
|
|
@@ -4,7 +4,7 @@ description: >
|
|
|
4
4
|
Use when detecting drift between CLAUDE.md (or AGENTS.md, the Codex CLI alias) / _meta narrative and
|
|
5
5
|
live repository state. Ten checks: absolute-path resolution, 01-projects/ count claims, issue-reference
|
|
6
6
|
freshness, session-file existence, command-count sync, session-config-parity (mandatory template keys =
|
|
7
|
-
error, opt-in gaps = warning), vault-dir-parity (CLAUDE.md vs AGENTS.md), generated-rule-staleness
|
|
7
|
+
error, opt-in gaps = note, unknown local keys = warning), vault-dir-parity (CLAUDE.md vs AGENTS.md), generated-rule-staleness
|
|
8
8
|
(WARN-only), rule-scoping (paths:/globs: frontmatter defects, dangling rule citations, zero-match
|
|
9
9
|
globs), and docs-parity (docs/components.md count-claims vs on-disk counts, template-vs-reference
|
|
10
10
|
config-key parity, stale .claude/metrics/ paths). Full per-check spec in the body table. Invoked as an
|
|
@@ -37,18 +37,18 @@ PHASE 1 IMPLEMENTED (2026-04-19). Session-end opt-in quality gate. Upstream of `
|
|
|
37
37
|
| 2 | `project-count-sync` | Hardcoded "N registered" / "N projects" claims next to `01-projects/` | compare to `ls -d 01-projects/*/` |
|
|
38
38
|
| 3 | `issue-reference-freshness` | `#NN` in forward-looking sections (What's Next, Backlog, Open Issues, Offene Themen, Todo, Next Steps) | `glab issue view NN --repo <origin>` |
|
|
39
39
|
| 4 | `session-file-existence` | `50-sessions/YYYY-MM-DD-*.md` references anywhere in scope | `existsSync(vault/50-sessions/<file>)` |
|
|
40
|
-
| 5 | `command-count` | "N commands" / "N /commands" claims in prose | compare to `
|
|
41
|
-
| 6 | `session-config-parity` | Top-level keys under `## Session Config` in `CLAUDE.md` / `AGENTS.md` | diff against `docs/session-config-template.md`; a missing MANDATORY (minimal-baseline) key is an error, a missing OPT-IN-baseline-only key is a warning |
|
|
40
|
+
| 5 | `command-count` | "N commands" / "N /commands" claims in prose | compare to the name-deduplicated union of `commands/*.md` and skills with explicit `user-invocable: true`; skipped only when both directories are absent |
|
|
41
|
+
| 6 | `session-config-parity` | Top-level keys under `## Session Config` in `CLAUDE.md` / `AGENTS.md` | diff against `docs/session-config-template.md`; a missing MANDATORY (minimal-baseline) key is an error, a missing OPT-IN-baseline-only key is a note, a local key absent from the template's Session Config blocks is a warning |
|
|
42
42
|
| 7 | `vault-dir-parity` | `vault-integration.vault-dir` in BOTH `CLAUDE.md` AND `AGENTS.md` | reuse `_parseVaultIntegration`; flag when the two files disagree. Short-circuits to PASS when the two are the SAME file by construction (symlink / same inode / byte-identical generated copy) |
|
|
43
43
|
| 8 | `generated-rule-staleness` *(WARN only)* | `.claude/rules/*.md` with `auto-generated: true` frontmatter | extract `learning-key`; WARN when the key is absent from `.orchestrator/metrics/learnings.jsonl` or its learning's `expires_at` is in the past; skipped silently when no auto-generated rules exist |
|
|
44
|
-
| 9 | `rule-scoping` | `.claude/rules/*.md` frontmatter + `## See Also` footers + `.claude/rules/<name>.md` citations in `CLAUDE.md`/`AGENTS.md` | six probes: `paths:` frontmatter (error), cited-but-missing rule citations (error), zero-match `globs:` patterns (warn), foreign PascalCase glob tokens (warn), unreadable rule files (warn — surfaced instead of silently skipped), declared fleet-intent globs (**note**, never warn); skipped silently when `.claude/rules/` is absent |
|
|
44
|
+
| 9 | `rule-scoping` | `.claude/rules/*.md` frontmatter + `## See Also` footers + `.claude/rules/<name>.md` citations in `CLAUDE.md`/`AGENTS.md` | six probes: `paths:` frontmatter incl. frontmatter not at line 1 (error, #1449), cited-but-missing rule citations (error), zero-match `globs:` patterns (warn), foreign PascalCase glob tokens (warn), unreadable rule files (warn — surfaced instead of silently skipped), declared fleet-intent globs (**note**, never warn); skipped silently when `.claude/rules/` is absent |
|
|
45
45
|
| 10 | `docs-parity` | `docs/components.md` count-claims, Session Config key parity (`docs/session-config-template.md` vs `docs/session-config-reference.md`), `.claude/metrics/` stale-path references in `docs/*.md` / `docs/examples/*.md` | three sub-checks (a/b/c) — see below; skipped silently when `docs/components.md` is absent |
|
|
46
46
|
|
|
47
47
|
Check 3 deliberately scopes to forward-looking sections. Mentions inside "Recently Closed", "Decisions", "Archive", etc. describe history and must not be flagged.
|
|
48
48
|
|
|
49
|
-
Check 5
|
|
49
|
+
Check 5 uses `countSlashCommands()` to count the union of non-hidden `*.md` files directly inside `commands/` and skills under `skills/` whose frontmatter explicitly declares `user-invocable: true`, deduplicated by name. The `commands/` directory is resolved relative to `VAULT_DIR` by default; `--commands-dir <path>` overrides that side of the union, while skills remain relative to `VAULT_DIR`. Either directory alone is sufficient; an existing directory with no commands contributes zero rather than causing a skip.
|
|
50
50
|
|
|
51
|
-
Check 6 (issue #30; severity split per #785) extracts the YAML block under `## Session Config` from both the canonical template (`docs/session-config-template.md` by default, override with `--config-template`) and the resolved local instruction file — reading the template TWICE, once per occurrence: `{ occurrence: 'first' }` reaches the "Full minimal baseline" (the 7 schema-mandatory keys), `{ occurrence: 'last' }` reaches the "Full opt-in baseline" (the full key catalog, a strict superset of the minimal block). A template key missing from the local file is a `session-config-parity` **error** when it belongs to the minimal-baseline set, and a **
|
|
51
|
+
Check 6 (issue #30; severity split per #785 and #1356) extracts the YAML block under `## Session Config` from both the canonical template (`docs/session-config-template.md` by default, override with `--config-template`) and the resolved local instruction file — reading the template TWICE, once per occurrence: `{ occurrence: 'first' }` reaches the "Full minimal baseline" (the 7 schema-mandatory keys), `{ occurrence: 'last' }` reaches the "Full opt-in baseline" (the full key catalog, a strict superset of the minimal block). A template key missing from the local file is a `session-config-parity` **error** when it belongs to the minimal-baseline set, and a **note** with `probe: "opt-in-gap"` when it is opt-in-baseline-only. A local key absent from those template blocks produces a **warning**: it may be a typo or a template gap, and this check makes no claim about documentation elsewhere or runtime use. When the template carries a single `## Session Config` heading (no separate opt-in block), `'first'` and `'last'` resolve to the same block, every template key is treated as mandatory, and unknown-local-key warnings are skipped. Both fenced YAML (```` ```yaml ... ``` ````) and raw YAML body (up to next `## ` heading) are accepted. The check skips gracefully when the template file is absent, when no instruction file is detected, or when explicitly disabled via `--skip-session-config-parity`.
|
|
52
52
|
|
|
53
53
|
The parity set is **template-driven**: every column-0 YAML key under `## Session Config` in the template (across both occurrences) is checked. As of the gsd Pattern Adoption Quick-Wins bundle (PRD 2026-05-22, issues #517–#521), the template-side keys include four opt-in top-level blocks:
|
|
54
54
|
|
|
@@ -57,7 +57,7 @@ The parity set is **template-driven**: every column-0 YAML key under `## Session
|
|
|
57
57
|
- `templates-first` (Pattern 3 / #519) — gh/glab template-read enforcement hook
|
|
58
58
|
- `verification-auto-fix` (Pattern 4 / #521) — opt-in auto-fix retry loop after Quality-Gate fail
|
|
59
59
|
|
|
60
|
-
A local CLAUDE.md / AGENTS.md that omits any of these
|
|
60
|
+
A local CLAUDE.md / AGENTS.md that omits any of these surfaces a `session-config-parity` **note** — they are opt-in-baseline-only keys, so omitting them never trips `mode: hard`, and never blocks `autonomous-gated` skill-evolution's `runConfigValidationGate()` (which runs the checker with `--mode hard` and reads only `errors[]`).
|
|
61
61
|
|
|
62
62
|
Check 7 (issue #600) is the **only** check that intentionally reads BOTH instruction files rather than the single alias-resolved one. The alias rule (CLAUDE.md wins ties, AGENTS.md is the Codex alias) means `resolveInstructionFile()` picks exactly one — so a repo carrying both files can silently let `AGENTS.md` drift out of sync with `CLAUDE.md`. A sibling project ran for weeks with a correct `vault-integration.vault-dir` in `CLAUDE.md` and a dead path in `AGENTS.md`. Check 7 reads `vault-integration.vault-dir` from each file (reusing the `_parseVaultIntegration` parser from `scripts/lib/config/vault-integration.mjs` — no hand-rolled YAML) and flags a `vault-dir-parity` error when the two values diverge (the error is attributed to `AGENTS.md`, the secondary alias, and names both values). The check skips gracefully when only one instruction file is present (nothing to compare), when neither file declares a `vault-integration:` block, or when explicitly disabled via `--skip-vault-dir-parity`. Two files that both omit `vault-dir` (both unset) agree and pass.
|
|
63
63
|
|
|
@@ -65,11 +65,11 @@ Check 7 (issue #600) is the **only** check that intentionally reads BOTH instruc
|
|
|
65
65
|
|
|
66
66
|
The same predicate guards Check 9's probe 2a, which iterates `['CLAUDE.md', 'AGENTS.md']`: with an aliased `AGENTS.md` it scans `CLAUDE.md` only, because otherwise every dangling `.claude/rules/*.md` citation would be reported TWICE — one defect, two errors, and in `--mode strict` a doubled blocking surface against a file the operator is not allowed to edit.
|
|
67
67
|
|
|
68
|
-
Check 9 (`rule-scoping`) validates `.claude/rules/*.md` frontmatter against the `scripts/lib/rule-loader.mjs` contract, catching the class of defect where a rule silently drifts out of the activation pipeline the loader actually implements. Six probes: **(1) paths-presence** — a
|
|
68
|
+
Check 9 (`rule-scoping`) validates `.claude/rules/*.md` frontmatter against the `scripts/lib/rule-loader.mjs` contract, catching the class of defect where a rule silently drifts out of the activation pipeline the loader actually implements. Six probes: **(1) paths-presence** — a frontmatter block that does not open on line 1 (text or a provenance comment before `---`) is an ERROR since #1449: Claude Code ignores it and loads the rule always-on; the probe reports `parseGlobsFrontmatter(...).defect === 'frontmatter-not-at-top'` with the remedy (move the block to line 1, provenance comment after the closing `---`). Otherwise, since #795 `paths:` is a full same-shape alias for `globs:` in `rule-loader.mjs` (`globs:` wins silently when both are present), so a well-formed `paths:`-only rule loads correctly SCOPED and is NOT flagged (#840). The probe fires only when the frontmatter textually carries a top-level `paths:` key AND `parseGlobsFrontmatter` — the loader's own parser, imported rather than re-derived — returns `globs === null` (e.g. a malformed frontmatter line makes it throw); that parse mismatch means the rule may silently load ALWAYS-ON, and is flagged as an error. **(2) cited-but-missing** — `(a)` `.claude/rules/<name>.md` citations inside `CLAUDE.md`/`AGENTS.md` that don't resolve to a file on disk, and `(b)` bare `<name>.md` tokens in a rule's own `## See Also` footer that don't exist as sibling rule files (tokens carrying a path separator, e.g. `../../skills/_shared/state-ownership.md`, are cross-directory references and explicitly out of scope); both flagged as errors. **(3) zero-match-globs** — a `globs:` pattern matching zero files in `git ls-files` (falls back to a manual directory walk when git is unavailable); flagged as a WARNING, not an error, because library/exemplar repos legitimately carry dead stack rules. **(4) foreign-glob** — a glob pattern containing a PascalCase product-like token (regex `[A-Z][a-z]+[A-Z]`, e.g. `WalkAITalkieTests`) — a likely copy-paste leftover from another project's rule scope; flagged as a WARNING. **(5) unreadable-file** — a rule file that could not be read (permissions, a race with a concurrent delete); flagged as a WARNING rather than silently skipped, so a completeness audit does not drop files, and so an unreadable file cannot brick the gate under `mode: hard`. **(6) fleet-intent-glob** — a `globs:` pattern DECLARED as fleet intent goes to `notes[]` (see below), never to `warnings[]`, and the declaration also settles probe 4 for that same pattern: a pattern the rule author declared is by construction not a copy-paste leftover. Glob matching reuses the same picomatch-with-inline-fallback resolution `scripts/lib/rule-loader.mjs` uses (`parseGlobsFrontmatter` is imported directly; the picomatch resolution itself is duplicated locally since `rule-loader.mjs` does not export a public matcher function). The check is skipped silently (no `checks_run` entry, no `checks_skipped` entry) when `.claude/rules/` is absent, or explicitly via `--skip-rule-scoping`.
|
|
69
69
|
|
|
70
70
|
**Fleet-intent declaration (Check 9, probe 6).** A rule may declare that a `globs:` pattern is aimed at CONSUMER repos rather than this one, via its own frontmatter key `fleet-intent-globs:`. Accepted forms are block style (`fleet-intent-globs:` plus indented ` - "<pattern>"` lines), flow style (`fleet-intent-globs: ["a", "b"]`) and a single inline value; surrounding quotes are stripped, blank and `#` comment lines are ignored. The key is inert for rule loading — `rule-loader.mjs`'s `parseGlobsFrontmatter` skips unknown top-level keys and their continuation lines. A declared pattern is reported in `notes[]` with its file, the pattern, whether it matches `≥1` or `0` tracked files here, and the declaration `source` (`frontmatter`). The grain is per pattern AND per rule file: declaring one glob amnesties neither a sibling glob in the same rule nor the same pattern in another rule. There is deliberately NO checker-side exemption list: a rule that does not declare the pattern itself is not exempt, so the declaration always sits with the rule it describes.
|
|
71
71
|
|
|
72
|
-
Check 10 (`docs-parity`, issue #780) validates the public docs surface against live repository state via three sub-checks, all reported under the single `docs-parity` check id: **(a) count-claims** — `docs/components.md`'s own heading counts ("## Skills (N user-facing)", "## Commands (N)", "## Agents (N typed sub-agents)", "## Hook event types (N)") are compared against the SAME actual on-disk derivation the surface-count family (Check 5) uses (`countSkills
|
|
72
|
+
Check 10 (`docs-parity`, issue #780) validates the public docs surface against live repository state via three sub-checks, all reported under the single `docs-parity` check id: **(a) count-claims** — `docs/components.md`'s own heading counts ("## Skills (N user-facing)", "## Commands (N)", "## Agents (N typed sub-agents)", "## Hook event types (N)") are compared against the SAME actual on-disk derivation the surface-count family (Check 5) uses (`countSkills`/`countSlashCommands`/`countAgents`/`readHookCounts`) — NOT Check 5's `claimRe` regexes, which are tuned for CLAUDE.md/README prose phrasing and verifiably do not match `components.md`'s own heading convention; a dedicated set of regexes tailored to the doc's actual authored structure is used instead. **(b) config-block-parity** — top-level Session Config keys documented in `docs/session-config-template.md` (opt-in baseline, via the Check 6 'last'-occurrence extractor) are compared against `docs/session-config-reference.md`; a key is considered documented in the reference when it appears in a `yaml` fence (any indent), a `##`/`###`/`####` heading, or the first cell of a markdown table row (`` | `key-name` | ... ``) — the reference's dominant documentation shape is one table row per field, so a naive fence-only extraction under-counts what it actually documents. Missing keys surface as `docs-parity` errors attributed to `docs/session-config-reference.md`. **(c) metrics-path-liveness** — any `.claude/metrics/` path reference (stale; canonical is `.orchestrator/metrics/`) found in root `docs/*.md` or `docs/examples/*.md` surfaces as an error at its exact file/line. The whole check is skipped silently (no `checks_run` entry, no `checks_skipped` entry) when `docs/components.md` is absent, or explicitly via `--skip-docs-parity`.
|
|
73
73
|
|
|
74
74
|
## Files
|
|
75
75
|
|
|
@@ -133,7 +133,7 @@ Environment:
|
|
|
133
133
|
}
|
|
134
134
|
```
|
|
135
135
|
|
|
136
|
-
`notes[]` is the THIRD output category, alongside `errors[]` and `warnings[]`: a finding that is reported but asks for no action. It never affects `status` or the exit code, and it is always present (empty array when nothing was noted, including on the short-circuit "no scope files matched" path).
|
|
136
|
+
`notes[]` is the THIRD output category, alongside `errors[]` and `warnings[]`: a finding that is reported but asks for no action. It never affects `status` or the exit code, and it is always present (empty array when nothing was noted, including on the short-circuit "no scope files matched" path). Producers are Check 6's `opt-in-gap` probe and Check 9's `fleet-intent-glob` probe. Opt-in-gap notes carry `check: "session-config-parity"`, `probe: "opt-in-gap"`, the instruction file and line, a message, and the omitted key in `extracted`; they have no `source` field.
|
|
137
137
|
|
|
138
138
|
The `resolved_path` / `resolved_kind` pair surfaces the alias resolution outcome (issue #33 AC2) so users on either platform can audit which instruction file the checker scanned. `kind: 'claude'` for `CLAUDE.md`, `kind: 'agents'` for `AGENTS.md`, `null` when neither was found.
|
|
139
139
|
|
|
@@ -25,7 +25,9 @@
|
|
|
25
25
|
* Reuses `_parseVaultIntegration` from scripts/lib/config/ for Check 7;
|
|
26
26
|
* reuses `parseGlobsFrontmatter` from scripts/lib/rule-loader.mjs for Check 9's
|
|
27
27
|
* glob extraction (mirrors the same picomatch-with-inline-fallback resolution
|
|
28
|
-
* rule-loader.mjs uses)
|
|
28
|
+
* rule-loader.mjs uses) and its line-1-only frontmatter rule (#1449: a block
|
|
29
|
+
* with anything before its opening `---` is not frontmatter, exactly as Claude
|
|
30
|
+
* Code reads it); otherwise pure Node stdlib.
|
|
29
31
|
*/
|
|
30
32
|
|
|
31
33
|
import { readFileSync, readdirSync, existsSync, statSync, lstatSync, realpathSync } from 'node:fs';
|
|
@@ -36,7 +38,7 @@ import { resolveInstructionFile } from '../../scripts/lib/common.mjs';
|
|
|
36
38
|
import { _parseVaultIntegration } from '../../scripts/lib/config/vault-integration.mjs';
|
|
37
39
|
import { _parseDriftCheck } from '../../scripts/lib/config/drift-check.mjs';
|
|
38
40
|
import { isSessionConfigHeading } from '../../scripts/lib/config/section-extractor.mjs';
|
|
39
|
-
import { parseGlobsFrontmatter } from '../../scripts/lib/rule-loader.mjs';
|
|
41
|
+
import { FRONTMATTER_NOT_AT_TOP, parseGlobsFrontmatter } from '../../scripts/lib/rule-loader.mjs';
|
|
40
42
|
import { resolveRepoSpec } from '../../scripts/lib/vcs-repo-spec.mjs';
|
|
41
43
|
import { userInvocableSkills } from '../../scripts/lib/user-invocable-skills.mjs';
|
|
42
44
|
|
|
@@ -573,7 +575,11 @@ function lookupIssueState(iid, repo, cache, vaultDir) {
|
|
|
573
575
|
// rule-loader.mjs's parseGlobsFrontmatter (the `paths:` alias, issue
|
|
574
576
|
// #795) fails to recognise — a genuine parse mismatch. A well-formed
|
|
575
577
|
// `paths:`-only rule is NOT flagged: since #795, `paths:` is a full
|
|
576
|
-
// alias for `globs:` and loads correctly SCOPED (#840).
|
|
578
|
+
// alias for `globs:` and loads correctly SCOPED (#840). Also an error
|
|
579
|
+
// (#1449): a frontmatter block that does not start on line 1 (a
|
|
580
|
+
// provenance comment, a blank line or any text before the opening
|
|
581
|
+
// `---`) — parseGlobsFrontmatter reports it as `frontmatter-not-at-top`,
|
|
582
|
+
// Claude Code ignores such a block, and the rule loads ALWAYS-ON.
|
|
577
583
|
// 2. cited-but-missing → errors[]: (a) `.claude/rules/<name>.md` citations in
|
|
578
584
|
// CLAUDE.md/AGENTS.md that don't exist on disk; (b) bare `<name>.md`
|
|
579
585
|
// tokens in a rule's own `## See Also` footer that don't exist on disk
|
|
@@ -674,21 +680,12 @@ function listTrackedFiles(vaultDir) {
|
|
|
674
680
|
}
|
|
675
681
|
}
|
|
676
682
|
|
|
677
|
-
const RULE_HEADER_LINE_RE = /^[ \t]*(?:<!--.*-->)?[ \t]*$/;
|
|
678
|
-
|
|
679
|
-
function stripLeadingRuleHeaderLines(content) {
|
|
680
|
-
const lines = content.split(/\r?\n/);
|
|
681
|
-
let idx = 0;
|
|
682
|
-
while (idx < lines.length && RULE_HEADER_LINE_RE.test(lines[idx])) idx++;
|
|
683
|
-
return lines.slice(idx).join('\n');
|
|
684
|
-
}
|
|
685
|
-
|
|
686
683
|
/** Extracts the raw frontmatter block body (text between the `---` delimiters),
|
|
687
|
-
* or null when the file has no frontmatter block.
|
|
688
|
-
*
|
|
689
|
-
*
|
|
684
|
+
* or null when the file has no frontmatter block. Only a block whose opening
|
|
685
|
+
* `---` is line 1 counts (#1449) — the rule rule-loader.mjs and Claude Code
|
|
686
|
+
* apply; a displaced block is body text, reported by probe 1 instead. */
|
|
690
687
|
function extractFrontmatterBlockBody(content) {
|
|
691
|
-
const m = /^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/.exec(
|
|
688
|
+
const m = /^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/.exec(content);
|
|
692
689
|
return m ? m[1] : null;
|
|
693
690
|
}
|
|
694
691
|
|
|
@@ -1173,7 +1170,7 @@ function main() {
|
|
|
1173
1170
|
// Minimal inline frontmatter extractor for auto-generated + learning-key.
|
|
1174
1171
|
// Reads the opening --- ... --- block from a markdown file.
|
|
1175
1172
|
function extractFrontmatterFields(mdContent) {
|
|
1176
|
-
const m = /^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/.exec(
|
|
1173
|
+
const m = /^---\r?\n([\s\S]*?)\r?\n---(?:\r?\n|$)/.exec(mdContent);
|
|
1177
1174
|
if (!m) return { autoGenerated: false, learningKey: null, expiresAt: null, evidenceDigest: null };
|
|
1178
1175
|
const block = m[1];
|
|
1179
1176
|
const autoGenM = /^auto-generated:\s*(.+)$/m.exec(block);
|
|
@@ -1370,10 +1367,21 @@ function main() {
|
|
|
1370
1367
|
// parseGlobsFrontmatter failed to recognise it (globs === null) —
|
|
1371
1368
|
// a genuine parse mismatch between this probe's textual detection
|
|
1372
1369
|
// and the loader's actual behaviour, not routine `paths:` usage.
|
|
1370
|
+
// #1449: the same parser also reports a frontmatter block that does
|
|
1371
|
+
// not start on line 1. Claude Code ignores that block, so whatever
|
|
1372
|
+
// key it carries (globs:, paths:, tier:, …) has no effect and the
|
|
1373
|
+
// rule loads ALWAYS-ON — an error of its own, reported instead of
|
|
1374
|
+
// the paths: mismatch (fmBody is null for such a file anyway).
|
|
1373
1375
|
let parsed;
|
|
1374
1376
|
try { parsed = parseGlobsFrontmatter(content); } catch { parsed = { globs: null, meta: {} }; }
|
|
1375
1377
|
const fmBody = extractFrontmatterBlockBody(content);
|
|
1376
|
-
if (
|
|
1378
|
+
if (parsed.defect === FRONTMATTER_NOT_AT_TOP) {
|
|
1379
|
+
errors.push({
|
|
1380
|
+
check: 'rule-scoping', file: relPath, line: 1,
|
|
1381
|
+
message: `${relPath}: frontmatter is not at line 1 (text before the opening ---) — Claude Code ignores it, the rule loads ALWAYS-ON; move the frontmatter to line 1 and put the provenance comment after the closing ---`,
|
|
1382
|
+
extracted: FRONTMATTER_NOT_AT_TOP,
|
|
1383
|
+
});
|
|
1384
|
+
} else if (fmBody && /^paths:/m.test(fmBody) && parsed.globs === null) {
|
|
1377
1385
|
errors.push({
|
|
1378
1386
|
check: 'rule-scoping', file: relPath, line: 1,
|
|
1379
1387
|
message: `Rule frontmatter declares 'paths:' but rule-loader.mjs's parseGlobsFrontmatter did not recognise it — the rule may silently load ALWAYS-ON. Verify the paths:/globs: frontmatter syntax.`,
|
|
@@ -19,7 +19,14 @@ Implemented from spec since upstream skill not yet published as of 2026-04-30.
|
|
|
19
19
|
|
|
20
20
|
## Invocation
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
`user-invocable: false` — there is no `/convergence-monitoring` slash command.
|
|
23
|
+
|
|
24
|
+
**Platform-native (primary, CC 2.1.105+):** registered as a plugin monitor
|
|
25
|
+
(`monitors/monitors.json`); auto-starts in the background on `wave-executor`
|
|
26
|
+
invocation and tails `.orchestrator/metrics/events.jsonl` for convergence signals.
|
|
27
|
+
|
|
28
|
+
**Direct/model invocation:** the model dispatches this skill by name for a
|
|
29
|
+
standalone assessment from current session history (fallback for harness < 2.1.105).
|
|
23
30
|
|
|
24
31
|
**Embedded:** Set `convergence-monitoring: true` in Session Config. `wave-executor` and
|
|
25
32
|
`/autoresearch` will call this skill automatically after each Impl-Core/Impl-Polish wave.
|
|
@@ -208,12 +208,56 @@ VEL_LINES_PRIOR=<read from events.jsonl for wave N-1>
|
|
|
208
208
|
|
|
209
209
|
The `velocity_drop` NDJSON signal counts **one `orchestrator.agent.stopped`
|
|
210
210
|
record per dispatched agent** (wave read from `wave` or `wave_number`) and fires
|
|
211
|
-
when the later wave stopped fewer agents than the earlier one.
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
211
|
+
when the later wave stopped fewer agents than the earlier one. `agent.stopped`
|
|
212
|
+
is the only per-agent record this repo actually emits with a wave number — but
|
|
213
|
+
the type is BIMODAL (#939/#949), so only records carrying a **non-empty `agent`
|
|
214
|
+
field** are counted; the rest are the phantom-stop class and counting them
|
|
215
|
+
inflated every per-wave count ~10.5x (#1379 P10).
|
|
216
|
+
|
|
217
|
+
Measured 2026-09-18 over `.orchestrator/metrics/events.jsonl` plus the legacy
|
|
218
|
+
`.jsonl.1` backup that then sat beside it — that backup was destroyed on
|
|
219
|
+
2026-09-19, so the counts below are a record, not something this repo's ledger
|
|
220
|
+
still reproduces.
|
|
221
|
+
|
|
222
|
+
**Read the ledger through `readEventsWithRotations(repoRoot, opts)`**
|
|
223
|
+
(`scripts/lib/events.mjs`), never by `cat`-ing rotation files together. It
|
|
224
|
+
returns the active file plus every archive in time order, counts unreadable
|
|
225
|
+
lines (`malformed_lines`), and — the reason it exists — reports a rotation
|
|
226
|
+
whose archive is no longer on disk as an entry in `gaps` with `complete:
|
|
227
|
+
false`, instead of silently returning a shorter history. Rotation itself no
|
|
228
|
+
longer writes a `.1`..`.N` ring: since #1401 it renames the active file to
|
|
229
|
+
`_archive/events-<firstTs>_<lastTs>.jsonl` and makes an
|
|
230
|
+
`orchestrator.events.rotated` record naming that archive the first line of the
|
|
231
|
+
new active file (`scripts/lib/events-rotation.mjs:230-258`); the reader still
|
|
232
|
+
reads a legacy ring left on disk by older versions
|
|
233
|
+
(`scripts/lib/events.mjs:512-539`).
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
node --input-type=module -e '
|
|
237
|
+
import { readEventsWithRotations } from "./scripts/lib/events.mjs";
|
|
238
|
+
const r = readEventsWithRotations(process.cwd());
|
|
239
|
+
console.error(JSON.stringify({ gaps: r.gaps, malformed_lines: r.malformed_lines, complete: r.complete }));
|
|
240
|
+
for (const e of r.events) process.stdout.write(JSON.stringify(e) + "\n");
|
|
241
|
+
' | jq -s '{
|
|
242
|
+
total_stopped: [.[]|select(.event=="orchestrator.agent.stopped")]|length,
|
|
243
|
+
with_wave: [.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null))]|length,
|
|
244
|
+
with_wave_no_agent: [.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null) and ((.agent//"")==""))]|length,
|
|
245
|
+
with_wave_and_agent:[.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null) and ((.agent//"")!=""))]|length }'
|
|
246
|
+
# 2026-09-18 (active + `.jsonl.1`):
|
|
247
|
+
# → { total_stopped: 16438, with_wave: 6553, with_wave_no_agent: 5930, with_wave_and_agent: 623 }
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
i.e. 5930 of 6553 wave-scoped stops (90.5%) are phantoms, against **0** records
|
|
251
|
+
of the `agent.dispatched` type the reader also accepts. The git-based
|
|
252
|
+
`VEL_COMMITS` / `VEL_LINES` recipe above is the SKILL-level computation and is
|
|
253
|
+
independent of the monitor.
|
|
254
|
+
|
|
255
|
+
The `console.error` line is the honesty half of the reading and belongs in every
|
|
256
|
+
re-measurement: a count taken from a ledger whose `gaps` is non-empty is a count
|
|
257
|
+
over an unknown fraction of the history. Its blind spot is bounded and named —
|
|
258
|
+
an archive deleted BEFORE #1401 left no tombstone and is undetectable by
|
|
259
|
+
construction (`scripts/lib/events.mjs:556-561`), which is exactly how the
|
|
260
|
+
53,896-line `.jsonl.1` vanished on 2026-09-19 without any analysis noticing.
|
|
217
261
|
|
|
218
262
|
### Thresholds
|
|
219
263
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: convergence-monitoring
|
|
3
|
-
user-invocable:
|
|
3
|
+
user-invocable: false
|
|
4
4
|
tags: [autoresearch, convergence, loop-control, wave-executor]
|
|
5
5
|
model: haiku
|
|
6
6
|
model-preference: sonnet
|
|
@@ -32,9 +32,17 @@ For harness < 2.1.105 (no monitor support), the skill's manual probes documented
|
|
|
32
32
|
|
|
33
33
|
## When to Invoke
|
|
34
34
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
35
|
+
This skill is `user-invocable: false` — there is no `/convergence-monitoring` slash
|
|
36
|
+
command. It is reached two ways:
|
|
37
|
+
|
|
38
|
+
**Platform-native monitor (primary path, CC 2.1.105+):** the background watcher below
|
|
39
|
+
auto-starts on `wave-executor` invocation and streams verdicts as it tails
|
|
40
|
+
`events.jsonl` — no explicit call needed.
|
|
41
|
+
|
|
42
|
+
**Direct/model invocation (fallback, harness < 2.1.105, or a standalone assessment):**
|
|
43
|
+
the model dispatches this skill by name — reads wave history from
|
|
44
|
+
`.orchestrator/metrics/events.jsonl`, computes all three signals, reports verdict +
|
|
45
|
+
recovery options.
|
|
38
46
|
|
|
39
47
|
**Embedded (loop context):** `/autoresearch` and `wave-executor` invoke this skill after
|
|
40
48
|
each Impl-Core or Impl-Polish wave when `convergence-monitoring: true` is set in Session
|
|
@@ -48,9 +56,10 @@ toward done or oscillating/stalling.
|
|
|
48
56
|
|
|
49
57
|
## Phase 0: Activation Gate
|
|
50
58
|
|
|
51
|
-
### 0.1
|
|
59
|
+
### 0.1 Direct Invocation
|
|
52
60
|
|
|
53
|
-
If invoked
|
|
61
|
+
If invoked directly (not embedded from `wave-executor` or `/autoresearch`), skip to
|
|
62
|
+
Phase 1 unconditionally.
|
|
54
63
|
|
|
55
64
|
### 0.2 Embedded Invocation
|
|
56
65
|
|
package/skills/eval/SKILL.md
CHANGED
|
@@ -15,7 +15,7 @@ args-schema:
|
|
|
15
15
|
- flag: --verify
|
|
16
16
|
description: "Re-evaluate a stored run-id and diff per-dimension for scoring drift (exit 1 on drift)"
|
|
17
17
|
description: >
|
|
18
|
-
Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-
|
|
18
|
+
Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-v2 dimensions, run /eval, evaluate this session, produce an eval report, or re-verify a stored eval run for reproducibility. Deterministic-first with an optional advisory LLM judge; never produces a global score.
|
|
19
19
|
---
|
|
20
20
|
|
|
21
21
|
> **Platform Note:** State files use the platform's native directory: `.claude/` (Claude Code), `.codex/` (Codex CLI), or `.cursor/` (Cursor IDE). Shared metrics + the eval journal live in `.orchestrator/metrics/`. See `skills/_shared/platform-tools.md`.
|
|
@@ -23,14 +23,17 @@ description: >
|
|
|
23
23
|
# Eval Skill — Session-Process Evaluation (aiat-llm-eval/1.0)
|
|
24
24
|
|
|
25
25
|
On-demand, honest measurement of ONE completed orchestrator session against the
|
|
26
|
-
pre-registered **rubric-
|
|
26
|
+
pre-registered **rubric-v2** check set. The deterministic engine
|
|
27
27
|
(`scripts/eval-session.mjs` → `scripts/lib/eval/engine.mjs`) reads only local
|
|
28
|
-
metrics files (`sessions.jsonl` + `events.jsonl`), scores the
|
|
28
|
+
metrics files (`sessions.jsonl` + `events.jsonl`), scores the six deterministic
|
|
29
29
|
dimensions, appends a `session-eval` record to the journal, and optionally
|
|
30
|
-
renders an HTML report. An opt-in LLM judge overlays
|
|
30
|
+
renders an HTML report. An opt-in LLM judge overlays ONE advisory dimension
|
|
31
|
+
(`instruction-adherence`; `report-quality` was retired in rubric-v2, #1381).
|
|
31
32
|
|
|
32
33
|
The standard this skill implements is [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md);
|
|
33
|
-
the frozen, content-hashed check set is [`skills/eval/rubric-
|
|
34
|
+
the frozen, content-hashed check set is [`skills/eval/rubric-v2.md`](./rubric-v2.md)
|
|
35
|
+
(stored records written before 2026-09-19 carry `rubric-v1` and are read against
|
|
36
|
+
[`rubric-v1.md`](./rubric-v1.md), which is never edited again).
|
|
34
37
|
|
|
35
38
|
## Invocation
|
|
36
39
|
|
|
@@ -43,7 +46,7 @@ Invoked as `/eval [--session <id>] [--no-write] [--verify <run-id>]` with argume
|
|
|
43
46
|
|
|
44
47
|
**On-demand `/eval` runs regardless of `eval.enabled`** — that flag gates only the automatic session-end eval phase (see Phase 1.1).
|
|
45
48
|
|
|
46
|
-
**Seams used:** `scripts/eval-session.mjs` (deterministic CLI) · `runEvalJudge` / `mergeJudgeDimensions` (`scripts/lib/eval/judge.mjs`, opt-in) · `writeEvalReport` (`scripts/lib/eval/report.mjs`) · `appendEvalRecord` (`scripts/lib/eval/sink.mjs`) · the `eval` config block · [`skills/eval/rubric-
|
|
49
|
+
**Seams used:** `scripts/eval-session.mjs` (deterministic CLI) · `runEvalJudge` / `mergeJudgeDimensions` (`scripts/lib/eval/judge.mjs`, opt-in) · `writeEvalReport` (`scripts/lib/eval/report.mjs`) · `appendEvalRecord` (`scripts/lib/eval/sink.mjs`) · the `eval` config block · [`skills/eval/rubric-v2.md`](./rubric-v2.md) (frozen check set).
|
|
47
50
|
|
|
48
51
|
## Posture Contract (load-bearing — read before executing)
|
|
49
52
|
|
|
@@ -52,9 +55,9 @@ Invoked as `/eval [--session <id>] [--no-write] [--verify <run-id>]` with argume
|
|
|
52
55
|
- **Never guess.** Missing source data yields `cannot-determine` (a first-class,
|
|
53
56
|
non-error verdict) with an honest reason — never a fabricated `pass`/`fail`.
|
|
54
57
|
Do NOT "fill in" a missing KPI or infer a gate result the events do not show.
|
|
55
|
-
- **Deterministic before judge.** The
|
|
56
|
-
on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated`
|
|
57
|
-
|
|
58
|
+
- **Deterministic before judge.** The six deterministic dimensions are complete
|
|
59
|
+
on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated` —
|
|
60
|
+
never blend a judge verdict into the deterministic tally.
|
|
58
61
|
- **Journal is SSOT; the report is a derived view.** The append-only
|
|
59
62
|
`.orchestrator/metrics/eval.jsonl` is authoritative. The HTML report is
|
|
60
63
|
rebuildable from any stored record and is never authoritative over the journal.
|
|
@@ -148,10 +151,13 @@ node scripts/eval-session.mjs [--session <id>] --json \
|
|
|
148
151
|
On exit `1` (e.g. "no completed session found"), surface the message and stop —
|
|
149
152
|
do not retry with fabricated inputs.
|
|
150
153
|
|
|
151
|
-
Parse the emitted JSON record. It carries `dimensions[]` (
|
|
152
|
-
entries
|
|
153
|
-
|
|
154
|
-
|
|
154
|
+
Parse the emitted JSON record. It carries `dimensions[]` (6 deterministic
|
|
155
|
+
entries — the two reported-only ones, `guard-friction` and `efficiency-kpis`,
|
|
156
|
+
are always `not-applicable`), `kpis{}`, `provenance.rubric_sha256` (the sha256
|
|
157
|
+
of `rubric-v2.md`; `null` means the rubric file was not found and the append
|
|
158
|
+
will fail validation), `model`, `harness`, and `run_id`. Unless `--no-write` was
|
|
159
|
+
passed, the record is already appended to `.orchestrator/metrics/eval.jsonl` by
|
|
160
|
+
the CLI.
|
|
155
161
|
|
|
156
162
|
**Contamination check:** if the human-render/summary reports a peer-overlapped
|
|
157
163
|
window, note it — `verification-evidence` and `gate-health` will read
|
|
@@ -187,9 +193,13 @@ const merged = mergeJudgeDimensions(record, dimensions);
|
|
|
187
193
|
appendEvalRecord(merged, { path: '.orchestrator/metrics/eval.jsonl' });
|
|
188
194
|
```
|
|
189
195
|
|
|
190
|
-
-
|
|
191
|
-
"uncalibrated"` (the schema firewall rejects any other shape). Keep
|
|
192
|
-
visibly separated from the deterministic
|
|
196
|
+
- The judge dimension arrives `advisory: true` + `calibration_status:
|
|
197
|
+
"uncalibrated"` (the schema firewall rejects any other shape). Keep it
|
|
198
|
+
visibly separated from the deterministic six in the summary.
|
|
199
|
+
- `runEvalJudge` pre-computes a `facts` block (`computeRecordFacts`) from the
|
|
200
|
+
deterministic evidence strings and hands it to the judge OUTSIDE the
|
|
201
|
+
untrusted-data fence — anything countable is counted in code, never inferred
|
|
202
|
+
by the model. Do not reimplement that here.
|
|
193
203
|
- If `runEvalJudge` returns a non-ok `status` (e.g. dispatch failed), keep the
|
|
194
204
|
deterministic record as-is and note the judge was unavailable — the
|
|
195
205
|
deterministic evaluation is complete without it.
|
|
@@ -226,20 +236,20 @@ const res = writeEvalReport(record, { generatedAt: new Date().toISOString() });
|
|
|
226
236
|
Emit a compact, honest per-dimension summary. Status lines only — no global score.
|
|
227
237
|
|
|
228
238
|
```
|
|
229
|
-
## /eval — <session_id> (self-evaluation, aiat-llm-eval/1.0 · rubric-
|
|
239
|
+
## /eval — <session_id> (self-evaluation, aiat-llm-eval/1.0 · rubric-v2 · n=1, no CI)
|
|
230
240
|
|
|
231
241
|
Deterministic:
|
|
232
242
|
verification-evidence PASS <one-line evidence>
|
|
233
243
|
plan-fidelity PASS completion_rate=1.0 (score)
|
|
234
244
|
gate-health PASS <one-line evidence>
|
|
235
|
-
process-safety PASS <one-line evidence + guard
|
|
245
|
+
process-safety PASS <one-line evidence + both guard disclosures>
|
|
246
|
+
guard-friction N/A (reported: blocked=… warned=… loop.warning=… <attribution>)
|
|
236
247
|
efficiency-kpis N/A (reported: duration=…s waves=… agents=… tok_in=… tok_out=… carryover=…)
|
|
237
248
|
|
|
238
249
|
Judge (advisory, uncalibrated) [only when eval.judge != off]:
|
|
239
|
-
instruction-adherence <verdict> advisory
|
|
240
|
-
report-quality <verdict> advisory
|
|
250
|
+
instruction-adherence <verdict> advisory (rule <n> applied)
|
|
241
251
|
|
|
242
|
-
cannot-determine: <k> of
|
|
252
|
+
cannot-determine: <k> of 6 deterministic dimensions (<reasons>)
|
|
243
253
|
Report: .orchestrator/eval/reports/<run_id>.html
|
|
244
254
|
Journal: .orchestrator/metrics/eval.jsonl (appended: <yes|--no-write>)
|
|
245
255
|
Re-verify: node scripts/eval-session.mjs --verify <run_id>
|
|
@@ -266,6 +276,11 @@ node scripts/eval-session.mjs --verify <run-id> --json
|
|
|
266
276
|
changed since the record was written — investigate, do not overwrite.
|
|
267
277
|
- `--verify` reproduces the stored model + timestamp verbatim (no env override),
|
|
268
278
|
so a MATCH is a real reproducibility proof of the scoring, not of model output.
|
|
279
|
+
- **A cross-version DRIFT is not a defect.** A stored `rubric-v1` record
|
|
280
|
+
re-scored by today's rubric-v2 engine necessarily differs on `process-safety`
|
|
281
|
+
and reports `present-in-fresh-only: guard-friction`. Read the record's
|
|
282
|
+
`rubric_version` before treating a diff as a regression (#1400 replaces that
|
|
283
|
+
report with an explicit version verdict).
|
|
269
284
|
|
|
270
285
|
---
|
|
271
286
|
|
|
@@ -282,9 +297,9 @@ platform. Only the judge phase needs harness-specific tooling.
|
|
|
282
297
|
`skills/_shared/platform-tools.md` § Agent Dispatch Pattern.
|
|
283
298
|
- **`harness.platform`** on the record is resolved from `$SO_PLATFORM`
|
|
284
299
|
(falls back to `claude-code`) inside the engine — no skill action needed.
|
|
285
|
-
- The deterministic
|
|
286
|
-
available on all platforms; the judge overlay is a Claude-Code-only
|
|
287
|
-
|
|
300
|
+
- The deterministic six dimensions + the HTML report + `--verify` are fully
|
|
301
|
+
available on all platforms; the judge overlay is a Claude-Code-only
|
|
302
|
+
enrichment.
|
|
288
303
|
|
|
289
304
|
---
|
|
290
305
|
|
package/skills/eval/rubric-v1.md
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
- **Conforms to standard:** `aiat-llm-eval/1.0` — see [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md)
|
|
6
6
|
- **Reference engine:** [`scripts/lib/eval/engine.mjs`](../../scripts/lib/eval/engine.mjs) (the executable scorers this document mirrors verbatim)
|
|
7
7
|
- **Hash binding:** the sha256 of THIS FILE is written to every record's `provenance.rubric_sha256`.
|
|
8
|
+
- **Superseded for NEW records by [`rubric-v2.md`](./rubric-v2.md) (2026-09-19, issue #1037).** This file stays frozen and authoritative for every record already carrying `rubric_version: "rubric-v1"` — its formulas are the ones those verdicts were produced under and must not be edited. v2 changed exactly two things: `process-safety` no longer fails on `destructive_guard.blocked`, and the guard counts moved to a new reported-only `guard-friction` dimension. The reasoning and the measurement live in rubric-v2 § Änderungen gegenüber v1.
|
|
8
9
|
|
|
9
10
|
> **Pre-Registration (leading principle, standard §1.1).** The checks below are
|
|
10
11
|
> **fixed BEFORE the first scored run executes against this rubric**. This document
|