session-orchestrator 5.2.0 → 5.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/architecture/SKILL.md +3 -1
- package/.agents/skills/autopilot/SKILL.md +5 -1
- package/.agents/skills/autopilot/agents/openai.yaml +5 -0
- package/.agents/skills/bootstrap/SKILL.md +5 -1
- package/.agents/skills/bootstrap/agents/openai.yaml +5 -0
- package/.agents/skills/brainstorm/SKILL.md +5 -1
- package/.agents/skills/brainstorm/agents/openai.yaml +5 -0
- package/.agents/skills/claude-md-drift-check/SKILL.md +3 -1
- package/.agents/skills/close/SKILL.md +5 -1
- package/.agents/skills/close/agents/openai.yaml +5 -0
- package/.agents/skills/convergence-monitoring/SKILL.md +4 -2
- package/.agents/skills/debug/SKILL.md +5 -1
- package/.agents/skills/debug/agents/openai.yaml +5 -0
- package/.agents/skills/discovery/SKILL.md +5 -1
- package/.agents/skills/discovery/agents/openai.yaml +5 -0
- package/.agents/skills/dispatcher/SKILL.md +5 -1
- package/.agents/skills/dispatcher/agents/openai.yaml +5 -0
- package/.agents/skills/docs-orchestrator/SKILL.md +3 -1
- package/.agents/skills/ecosystem-health/SKILL.md +3 -1
- package/.agents/skills/eli5/SKILL.md +5 -1
- package/.agents/skills/eli5/agents/openai.yaml +5 -0
- package/.agents/skills/eval/SKILL.md +6 -2
- package/.agents/skills/eval/agents/openai.yaml +5 -0
- package/.agents/skills/evolve/SKILL.md +6 -2
- package/.agents/skills/evolve/agents/openai.yaml +5 -0
- package/.agents/skills/frontmatter-guard/SKILL.md +3 -1
- package/.agents/skills/gitlab-ops/SKILL.md +3 -1
- package/.agents/skills/gitlab-portfolio/SKILL.md +3 -1
- package/.agents/skills/go/SKILL.md +5 -1
- package/.agents/skills/go/agents/openai.yaml +5 -0
- package/.agents/skills/grill/SKILL.md +5 -1
- package/.agents/skills/grill/agents/openai.yaml +5 -0
- package/.agents/skills/harness-audit/SKILL.md +5 -1
- package/.agents/skills/harness-audit/agents/openai.yaml +5 -0
- package/.agents/skills/hook-development/SKILL.md +3 -1
- package/.agents/skills/mcp-builder/SKILL.md +3 -1
- package/.agents/skills/memory-cleanup/SKILL.md +5 -1
- package/.agents/skills/memory-cleanup/agents/openai.yaml +5 -0
- package/.agents/skills/mode-selector/SKILL.md +3 -1
- package/.agents/skills/npm-publish/SKILL.md +4 -2
- package/.agents/skills/peekaboo-driver/SKILL.md +3 -1
- package/.agents/skills/persona-panel/SKILL.md +5 -1
- package/.agents/skills/persona-panel/agents/openai.yaml +5 -0
- package/.agents/skills/plan/SKILL.md +5 -1
- package/.agents/skills/plan/agents/openai.yaml +5 -0
- package/.agents/skills/playwright-driver/SKILL.md +3 -1
- package/.agents/skills/portfolio/SKILL.md +5 -1
- package/.agents/skills/portfolio/agents/openai.yaml +5 -0
- package/.agents/skills/quality-gates/SKILL.md +3 -1
- package/.agents/skills/reconcile/SKILL.md +5 -1
- package/.agents/skills/reconcile/agents/openai.yaml +5 -0
- package/.agents/skills/release/SKILL.md +5 -1
- package/.agents/skills/release/agents/openai.yaml +5 -0
- package/.agents/skills/remote-offload/SKILL.md +3 -1
- package/.agents/skills/repo-audit/SKILL.md +5 -1
- package/.agents/skills/repo-audit/agents/openai.yaml +5 -0
- package/.agents/skills/session/SKILL.md +21 -0
- package/.agents/skills/session/agents/openai.yaml +5 -0
- package/.agents/skills/session-end/SKILL.md +3 -1
- package/.agents/skills/session-plan/SKILL.md +3 -1
- package/.agents/skills/session-start/SKILL.md +3 -1
- package/.agents/skills/spinout/SKILL.md +5 -1
- package/.agents/skills/spinout/agents/openai.yaml +5 -0
- package/.agents/skills/sunset-review/SKILL.md +5 -1
- package/.agents/skills/sunset-review/agents/openai.yaml +5 -0
- package/.agents/skills/templates-ack/SKILL.md +21 -0
- package/.agents/skills/templates-ack/agents/openai.yaml +5 -0
- package/.agents/skills/test/SKILL.md +5 -1
- package/.agents/skills/test/agents/openai.yaml +5 -0
- package/.agents/skills/test-runner/SKILL.md +3 -1
- package/.agents/skills/tmux-layout/SKILL.md +3 -1
- package/.agents/skills/using-orchestrator/SKILL.md +3 -1
- package/.agents/skills/ux-grill/SKILL.md +5 -1
- package/.agents/skills/ux-grill/agents/openai.yaml +5 -0
- package/.agents/skills/vault-mirror/SKILL.md +3 -1
- package/.agents/skills/vault-sync/SKILL.md +3 -1
- package/.agents/skills/wave-executor/SKILL.md +3 -1
- package/.agents/skills/write-executable-plan/SKILL.md +3 -1
- package/.claude-plugin/marketplace.json +1 -1
- package/.claude-plugin/plugin.json +1 -1
- package/.codex-plugin/plugin.json +4 -4
- package/.codex-plugin/skills/convergence-monitoring/SKILL.md +1 -3
- package/.codex-plugin/skills/eval/SKILL.md +1 -1
- package/.codex-plugin/skills/evolve/SKILL.md +1 -1
- package/.codex-plugin/skills/npm-publish/SKILL.md +1 -3
- package/.codex-plugin/skills/session/SKILL.md +1 -1
- package/.cursor/commands/eval.md +1 -1
- package/.cursor/commands/session.md +1 -1
- package/.cursor/rules/000-session-orchestrator.mdc +0 -2
- package/.cursor/rules/050-plan.mdc +1 -1
- package/.cursor/skills/convergence-monitoring/SKILL.md +1 -0
- package/.cursor/skills/eval/SKILL.md +1 -1
- package/.cursor/skills/npm-publish/SKILL.md +1 -0
- package/.cursor-plugin/plugin.json +1 -1
- package/.orchestrator/policy/blocked-commands.json +12 -3
- package/AGENTS.md +3 -2
- package/CHANGELOG.md +136 -0
- package/README.md +9 -9
- package/SECURITY.md +12 -0
- package/agents/dialectic-deriver.md +13 -10
- package/agents/eval-judge.md +67 -45
- package/agents/skill-applied-judge.md +34 -19
- package/commands/session.md +7 -3
- package/docs/baseline.md +12 -6
- package/docs/codex-setup.md +14 -2
- package/docs/components.md +7 -5
- package/docs/events-schema.md +56 -9
- package/docs/rule-authoring.md +58 -6
- package/docs/session-config-reference.md +100 -7
- package/docs/session-config-template.md +31 -2
- package/docs/telemetry.md +2 -0
- package/hooks/_lib/hook-import-set.json +85 -8
- package/hooks/_lib/subagent-transcript.mjs +582 -31
- package/hooks/config-protection.mjs +11 -3
- package/hooks/cwd-change-restore.mjs +11 -3
- package/hooks/enforce-commands.mjs +70 -23
- package/hooks/enforce-scope.mjs +143 -33
- package/hooks/hooks-codex.json +1 -1
- package/hooks/hooks.json +1 -1
- package/hooks/loop-guard.mjs +11 -3
- package/hooks/on-session-end.mjs +58 -23
- package/hooks/on-session-start.mjs +48 -11
- package/hooks/on-stop.mjs +168 -22
- package/hooks/operator-steer.mjs +11 -3
- package/hooks/post-bash-issue-budget-refund.mjs +18 -8
- package/hooks/post-bash-write-verify.mjs +3 -2
- package/hooks/post-edit-import-probe.mjs +17 -9
- package/hooks/post-edit-validate.mjs +13 -5
- package/hooks/post-subagent-discovery-validator.mjs +98 -13
- package/hooks/post-tool-batch-wave-signal.mjs +200 -38
- package/hooks/post-tool-failure-corrective-context.mjs +11 -5
- package/hooks/post-tooluse-frontend-slop.mjs +10 -4
- package/hooks/pre-auq-clarity.mjs +15 -2
- package/hooks/pre-bash-destructive-guard.mjs +80 -9
- package/hooks/pre-bash-issue-budget.mjs +16 -11
- package/hooks/pre-bash-memory-propose-audit.mjs +86 -54
- package/hooks/pre-bash-sessions-ledger-guard.mjs +391 -20
- package/hooks/pre-bash-staging-fence.mjs +335 -31
- package/hooks/pre-bash-templates-first.mjs +19 -14
- package/hooks/pre-task-scope-disjoint.mjs +233 -2
- package/hooks/subagent-telemetry.mjs +15 -19
- package/hooks/wave-scope-commit-guard.mjs +197 -100
- package/monitors/monitors.json +1 -1
- package/output-styles/wave-summary.md +1 -1
- package/package.json +1 -1
- package/pi/prompts/eval.md +1 -1
- package/pi/prompts/session.md +1 -1
- package/rules/README.md +1 -1
- package/rules/opt-in-domain/prompt-caching.md +1 -1
- package/rules/opt-in-stack/backend-data.md +1 -1
- package/rules/opt-in-stack/backend.md +3 -3
- package/rules/opt-in-stack/frontend.md +1 -1
- package/rules/opt-in-stack/security-web.md +3 -3
- package/rules/opt-in-stack/swift.md +1 -1
- package/scripts/autopilot.mjs +23 -2
- package/scripts/backfill-abandoned-sessions.mjs +117 -15
- package/scripts/check-sessions-integrity.mjs +300 -0
- package/scripts/dialectic-deriver.mjs +50 -13
- package/scripts/emit-session.mjs +75 -29
- package/scripts/eval-session.mjs +65 -3
- package/scripts/generate-agents-skills.mjs +102 -29
- package/scripts/generate-cursor-adapter.mjs +61 -16
- package/scripts/lib/agent-status.mjs +2 -31
- package/scripts/lib/auq/clarity.mjs +10 -2
- package/scripts/lib/auq/parse.mjs +12 -31
- package/scripts/lib/auq/schema.mjs +56 -41
- package/scripts/lib/auto-dialectic.mjs +304 -15
- package/scripts/lib/autopilot/flags.mjs +12 -1
- package/scripts/lib/autopilot/kill-switches.mjs +6 -3
- package/scripts/lib/autopilot/loop.mjs +14 -1
- package/scripts/lib/autopilot/stall-sampler.mjs +80 -23
- package/scripts/lib/ci-status-banner.mjs +376 -16
- package/scripts/lib/command-blocker.mjs +275 -28
- package/scripts/lib/config/dialectic.mjs +12 -3
- package/scripts/lib/config/gate.mjs +74 -0
- package/scripts/lib/config/reaper.mjs +162 -0
- package/scripts/lib/config.mjs +14 -0
- package/scripts/lib/convergence-monitor.mjs +74 -11
- package/scripts/lib/ecosystem-health.mjs +11 -0
- package/scripts/lib/eval/engine.mjs +421 -53
- package/scripts/lib/eval/judge.mjs +463 -40
- package/scripts/lib/eval/schema.mjs +10 -1
- package/scripts/lib/events-rotation.mjs +221 -25
- package/scripts/lib/events-schema.mjs +114 -0
- package/scripts/lib/events.mjs +524 -5
- package/scripts/lib/frontmatter-guard.mjs +21 -10
- package/scripts/lib/gates/gate-baseline.mjs +27 -2
- package/scripts/lib/gates/gate-full.mjs +28 -3
- package/scripts/lib/gates/gate-helpers.mjs +243 -21
- package/scripts/lib/gates/gate-incremental.mjs +28 -3
- package/scripts/lib/gates/gate-per-file.mjs +27 -2
- package/scripts/lib/gitlab-portfolio/markdown-writer.mjs +6 -1
- package/scripts/lib/instruction-budget-guard.mjs +146 -4
- package/scripts/lib/io.mjs +42 -8
- package/scripts/lib/issue-close-strip-labels.mjs +207 -49
- package/scripts/lib/js-mask.mjs +197 -0
- package/scripts/lib/learnings/evolve-telemetry.mjs +11 -7
- package/scripts/lib/maintenance-due-banner.mjs +53 -88
- package/scripts/lib/orphan-reaper.mjs +1588 -0
- package/scripts/lib/peer-cards/merger.mjs +48 -10
- package/scripts/lib/peer-cards/reader.mjs +78 -2
- package/scripts/lib/process-group.mjs +899 -0
- package/scripts/lib/quality-gate.mjs +107 -28
- package/scripts/lib/reconcile/backlog.mjs +368 -0
- package/scripts/lib/reconcile/engine.mjs +55 -188
- package/scripts/lib/reconcile/rule-expiry-sweep.mjs +302 -60
- package/scripts/lib/reconcile/sanitize.mjs +69 -3
- package/scripts/lib/reconcile-nudge-banner.mjs +138 -45
- package/scripts/lib/resource-probe/parsers.mjs +31 -0
- package/scripts/lib/rule-loader.mjs +41 -12
- package/scripts/lib/scope-echo.mjs +39 -2
- package/scripts/lib/scope-gate.mjs +605 -1
- package/scripts/lib/session-close-backfill.mjs +33 -6
- package/scripts/lib/session-id.mjs +9 -20
- package/scripts/lib/session-invocation.mjs +20 -0
- package/scripts/lib/session-schema/constants.mjs +30 -2
- package/scripts/lib/session-schema/normalizer.mjs +56 -4
- package/scripts/lib/session-schema.mjs +8 -3
- package/scripts/lib/session-start-probes.mjs +95 -10
- package/scripts/lib/sessions-canonical.mjs +23 -0
- package/scripts/lib/sessions-integrity-banner.mjs +7 -1
- package/scripts/lib/sessions-staleness-banner.mjs +193 -51
- package/scripts/lib/skill-evidence-window.mjs +891 -0
- package/scripts/lib/skill-evolution/candidate-intake.mjs +133 -12
- package/scripts/lib/skill-evolution/engine.mjs +18 -9
- package/scripts/lib/skill-judge.mjs +45 -3
- package/scripts/lib/tail-window.mjs +56 -0
- package/scripts/lib/telemetry/schema.mjs +30 -0
- package/scripts/lib/telemetry/sync.mjs +61 -6
- package/scripts/lib/telemetry-flush-health-banner.mjs +4 -22
- package/scripts/lib/test-runner/issue-reconcile.mjs +48 -16
- package/scripts/lib/tmux-layout/telemetry-stats.mjs +72 -13
- package/scripts/lib/user-invocable-skills.mjs +23 -3
- package/scripts/lib/ux-grill/reconcile.mjs +48 -22
- package/scripts/lib/validate/check-agents-skills.mjs +26 -15
- package/scripts/lib/validate/check-cursor-adapter.mjs +1 -0
- package/scripts/lib/validate/check-entry-guard.mjs +13 -50
- package/scripts/lib/validate/check-hook-entry-guards.mjs +636 -0
- package/scripts/lib/validate/check-pi-prompts.mjs +1 -0
- package/scripts/lib/validate/check-rules.mjs +7 -5
- package/scripts/lib/validate/check-skill-links.mjs +9 -1
- package/scripts/lib/validate/check-skill-script-paths.mjs +239 -27
- package/scripts/lib/validate/check-test-git-config-target.mjs +24 -34
- package/scripts/lib/validate/check-untracked-test-deps.mjs +7 -102
- package/scripts/lib/validate/check-unwired-features.mjs +130 -27
- package/scripts/lib/validate/check-validator-registration.mjs +34 -10
- package/scripts/lib/validate/confidential-names.mjs +10 -0
- package/scripts/lib/validate-vendored-rules.mjs +4 -3
- package/scripts/lib/vault-mirror/namespace.mjs +46 -8
- package/scripts/lib/vault-mirror/process.mjs +10 -3
- package/scripts/lib/vault-mirror/render-sessions.mjs +12 -2
- package/scripts/lib/vault-status/narrative-mirror.mjs +31 -7
- package/scripts/lib/vault-yaml.mjs +118 -0
- package/scripts/lib/worktree/lifecycle.mjs +153 -1
- package/scripts/release-session-lock.mjs +305 -0
- package/scripts/release.mjs +30 -5
- package/scripts/resolve-session-invocation.mjs +59 -0
- package/scripts/run-quality-gate.mjs +156 -17
- package/scripts/sweep-expired-rules.mjs +14 -3
- package/scripts/validate-plugin.mjs +12 -0
- package/scripts/validate-wave-scope.mjs +32 -105
- package/scripts/vault-mirror.mjs +9 -1
- package/skills/_shared/platform-tools.md +23 -11
- package/skills/autopilot/SKILL.md +22 -7
- package/skills/claude-md-drift-check/SKILL.md +1 -1
- package/skills/convergence-monitoring/README.md +8 -1
- package/skills/convergence-monitoring/SIGNALS.md +50 -6
- package/skills/convergence-monitoring/SKILL.md +15 -6
- package/skills/eval/SKILL.md +39 -24
- package/skills/eval/rubric-v1.md +1 -0
- package/skills/eval/rubric-v2.md +457 -0
- package/skills/evolve/SKILL.md +1 -1
- package/skills/evolve/references/evolve-dialectic-mode.md +42 -25
- package/skills/gitlab-ops/SKILL.md +3 -2
- package/skills/npm-publish/SKILL.md +1 -1
- package/skills/reconcile/SKILL.md +11 -0
- package/skills/session-end/SKILL.md +13 -16
- package/skills/session-end/discovery-scan.md +1 -1
- package/skills/session-end/phase-3-6-tail.md +55 -9
- package/skills/session-end/references/phase-5-issue-cleanup.md +9 -14
- package/skills/session-end/session-metrics-write.md +10 -0
- package/skills/session-plan/SKILL.md +17 -5
- package/skills/session-plan/references/session-plan-task-classification.md +2 -2
- package/skills/session-start/references/phase-4-ssot-environment-check.md +2 -1
- package/skills/ux-grill/SKILL.md +1 -1
- package/skills/wave-executor/SKILL.md +8 -4
- package/skills/wave-executor/circuit-breaker.md +2 -0
- package/skills/wave-executor/references/wave-executor-state-init.md +5 -3
- package/skills/wave-executor/references/wave-loop-dispatch.md +2 -1
- package/.codex-plugin/skills/convergence-monitoring/agents/openai.yaml +0 -5
- package/.codex-plugin/skills/npm-publish/agents/openai.yaml +0 -5
- package/.cursor/commands/convergence-monitoring.md +0 -13
- package/.cursor/commands/npm-publish.md +0 -13
- package/pi/prompts/convergence-monitoring.md +0 -11
- package/pi/prompts/npm-publish.md +0 -11
|
@@ -235,6 +235,7 @@ import { runLoop, parseFlags } from '$PLUGIN_ROOT/scripts/lib/autopilot.mjs';
|
|
|
235
235
|
import { buildLiveSignals } from '$PLUGIN_ROOT/scripts/lib/build-live-signals.mjs';
|
|
236
236
|
import { selectMode } from '$PLUGIN_ROOT/scripts/lib/mode-selector.mjs';
|
|
237
237
|
import { probe, evaluate } from '$PLUGIN_ROOT/scripts/lib/resource-probe.mjs';
|
|
238
|
+
import { detectPeers } from '$PLUGIN_ROOT/scripts/lib/session-registry.mjs';
|
|
238
239
|
|
|
239
240
|
const flags = parseFlags(process.argv.slice(2));
|
|
240
241
|
|
|
@@ -250,14 +251,27 @@ const modeSelector = async () => {
|
|
|
250
251
|
return selectMode(signals);
|
|
251
252
|
};
|
|
252
253
|
|
|
254
|
+
let cachedProbeSnapshot = null; // written by peerCounter, read by resourceEvaluator
|
|
255
|
+
|
|
253
256
|
const resourceEvaluator = () => {
|
|
254
|
-
|
|
255
|
-
|
|
257
|
+
// Synchronous by contract: never calls probe() itself (it is async) — reads
|
|
258
|
+
// the snapshot peerCounter refreshed on the prior iteration.
|
|
259
|
+
if (cachedProbeSnapshot === null) {
|
|
260
|
+
return { verdict: 'warn', reasons: ['probe not yet available'], recommended_agents_per_wave_cap: null };
|
|
261
|
+
}
|
|
262
|
+
return evaluate(cachedProbeSnapshot, thresholds);
|
|
256
263
|
};
|
|
257
264
|
|
|
258
|
-
const peerCounter = () => {
|
|
259
|
-
//
|
|
260
|
-
|
|
265
|
+
const peerCounter = async () => {
|
|
266
|
+
// A SESSION count, not a process count (host-resources.md HR-103) —
|
|
267
|
+
// claude_processes_count runs ~6 processes per session. `ownSessionId` is
|
|
268
|
+
// this coordinator's registry id, so detectPeers() excludes it.
|
|
269
|
+
const [peers, snapshot] = await Promise.all([
|
|
270
|
+
detectPeers({ sessionId: ownSessionId, freshnessMin: 15 }),
|
|
271
|
+
probe(),
|
|
272
|
+
]);
|
|
273
|
+
cachedProbeSnapshot = snapshot;
|
|
274
|
+
return peers.length;
|
|
261
275
|
};
|
|
262
276
|
|
|
263
277
|
const sessionRunner = async ({ mode, autopilotRunId }) => {
|
|
@@ -289,8 +303,9 @@ The in-process driver has Claude (the coordinator) call `/session <mode>` betwee
|
|
|
289
303
|
- **Pro:** zero new infra. Reuses canonical kill-switch logic. Validates `buildLiveSignals`
|
|
290
304
|
against real Phase 7.5 swap before headless complexity. Each iteration carries
|
|
291
305
|
inter-session memory through STATE.md / sessions.jsonl / learnings.
|
|
292
|
-
- **Con:** not truly autonomous — Claude must stay in the chat.
|
|
293
|
-
walk-away UX
|
|
306
|
+
- **Con:** not truly autonomous — Claude must stay in the chat. The in-process
|
|
307
|
+
driver does not deliver walk-away UX; that is the shipped headless driver
|
|
308
|
+
(`--headless`, `scripts/autopilot.mjs`, Phase C-5; see § Headless Driver Wiring).
|
|
294
309
|
|
|
295
310
|
### Headless Driver Wiring (Option A — `scripts/autopilot.mjs`)
|
|
296
311
|
|
|
@@ -65,7 +65,7 @@ Check 7 (issue #600) is the **only** check that intentionally reads BOTH instruc
|
|
|
65
65
|
|
|
66
66
|
The same predicate guards Check 9's probe 2a, which iterates `['CLAUDE.md', 'AGENTS.md']`: with an aliased `AGENTS.md` it scans `CLAUDE.md` only, because otherwise every dangling `.claude/rules/*.md` citation would be reported TWICE — one defect, two errors, and in `--mode strict` a doubled blocking surface against a file the operator is not allowed to edit.
|
|
67
67
|
|
|
68
|
-
Check 9 (`rule-scoping`) validates `.claude/rules/*.md` frontmatter against the `scripts/lib/rule-loader.mjs` contract, catching the class of defect where a rule silently drifts out of the activation pipeline the loader actually implements. Six probes: **(1) paths-presence** —
|
|
68
|
+
Check 9 (`rule-scoping`) validates `.claude/rules/*.md` frontmatter against the `scripts/lib/rule-loader.mjs` contract, catching the class of defect where a rule silently drifts out of the activation pipeline the loader actually implements. Six probes: **(1) paths-presence** — since #795 `paths:` is a full same-shape alias for `globs:` in `rule-loader.mjs` (`globs:` wins silently when both are present), so a well-formed `paths:`-only rule loads correctly SCOPED and is NOT flagged (#840). The probe fires only when the frontmatter textually carries a top-level `paths:` key AND `parseGlobsFrontmatter` — the loader's own parser, imported rather than re-derived — returns `globs === null` (e.g. a malformed frontmatter line makes it throw); that parse mismatch means the rule may silently load ALWAYS-ON, and is flagged as an error. **(2) cited-but-missing** — `(a)` `.claude/rules/<name>.md` citations inside `CLAUDE.md`/`AGENTS.md` that don't resolve to a file on disk, and `(b)` bare `<name>.md` tokens in a rule's own `## See Also` footer that don't exist as sibling rule files (tokens carrying a path separator, e.g. `../../skills/_shared/state-ownership.md`, are cross-directory references and explicitly out of scope); both flagged as errors. **(3) zero-match-globs** — a `globs:` pattern matching zero files in `git ls-files` (falls back to a manual directory walk when git is unavailable); flagged as a WARNING, not an error, because library/exemplar repos legitimately carry dead stack rules. **(4) foreign-glob** — a glob pattern containing a PascalCase product-like token (regex `[A-Z][a-z]+[A-Z]`, e.g. `WalkAITalkieTests`) — a likely copy-paste leftover from another project's rule scope; flagged as a WARNING. **(5) unreadable-file** — a rule file that could not be read (permissions, a race with a concurrent delete); flagged as a WARNING rather than silently skipped, so a completeness audit does not drop files, and so an unreadable file cannot brick the gate under `mode: hard`. **(6) fleet-intent-glob** — a `globs:` pattern DECLARED as fleet intent goes to `notes[]` (see below), never to `warnings[]`, and the declaration also settles probe 4 for that same pattern: a pattern the rule author declared is by construction not a copy-paste leftover. Glob matching reuses the same picomatch-with-inline-fallback resolution `scripts/lib/rule-loader.mjs` uses (`parseGlobsFrontmatter` is imported directly; the picomatch resolution itself is duplicated locally since `rule-loader.mjs` does not export a public matcher function). The check is skipped silently (no `checks_run` entry, no `checks_skipped` entry) when `.claude/rules/` is absent, or explicitly via `--skip-rule-scoping`.
|
|
69
69
|
|
|
70
70
|
**Fleet-intent declaration (Check 9, probe 6).** A rule may declare that a `globs:` pattern is aimed at CONSUMER repos rather than this one, via its own frontmatter key `fleet-intent-globs:`. Accepted forms are block style (`fleet-intent-globs:` plus indented ` - "<pattern>"` lines), flow style (`fleet-intent-globs: ["a", "b"]`) and a single inline value; surrounding quotes are stripped, blank and `#` comment lines are ignored. The key is inert for rule loading — `rule-loader.mjs`'s `parseGlobsFrontmatter` skips unknown top-level keys and their continuation lines. A declared pattern is reported in `notes[]` with its file, the pattern, whether it matches `≥1` or `0` tracked files here, and the declaration `source` (`frontmatter`). The grain is per pattern AND per rule file: declaring one glob amnesties neither a sibling glob in the same rule nor the same pattern in another rule. There is deliberately NO checker-side exemption list: a rule that does not declare the pattern itself is not exempt, so the declaration always sits with the rule it describes.
|
|
71
71
|
|
|
@@ -19,7 +19,14 @@ Implemented from spec since upstream skill not yet published as of 2026-04-30.
|
|
|
19
19
|
|
|
20
20
|
## Invocation
|
|
21
21
|
|
|
22
|
-
|
|
22
|
+
`user-invocable: false` — there is no `/convergence-monitoring` slash command.
|
|
23
|
+
|
|
24
|
+
**Platform-native (primary, CC 2.1.105+):** registered as a plugin monitor
|
|
25
|
+
(`monitors/monitors.json`); auto-starts in the background on `wave-executor`
|
|
26
|
+
invocation and tails `.orchestrator/metrics/events.jsonl` for convergence signals.
|
|
27
|
+
|
|
28
|
+
**Direct/model invocation:** the model dispatches this skill by name for a
|
|
29
|
+
standalone assessment from current session history (fallback for harness < 2.1.105).
|
|
23
30
|
|
|
24
31
|
**Embedded:** Set `convergence-monitoring: true` in Session Config. `wave-executor` and
|
|
25
32
|
`/autoresearch` will call this skill automatically after each Impl-Core/Impl-Polish wave.
|
|
@@ -208,12 +208,56 @@ VEL_LINES_PRIOR=<read from events.jsonl for wave N-1>
|
|
|
208
208
|
|
|
209
209
|
The `velocity_drop` NDJSON signal counts **one `orchestrator.agent.stopped`
|
|
210
210
|
record per dispatched agent** (wave read from `wave` or `wave_number`) and fires
|
|
211
|
-
when the later wave stopped fewer agents than the earlier one.
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
211
|
+
when the later wave stopped fewer agents than the earlier one. `agent.stopped`
|
|
212
|
+
is the only per-agent record this repo actually emits with a wave number — but
|
|
213
|
+
the type is BIMODAL (#939/#949), so only records carrying a **non-empty `agent`
|
|
214
|
+
field** are counted; the rest are the phantom-stop class and counting them
|
|
215
|
+
inflated every per-wave count ~10.5x (#1379 P10).
|
|
216
|
+
|
|
217
|
+
Measured 2026-09-18 over `.orchestrator/metrics/events.jsonl` plus the legacy
|
|
218
|
+
`.jsonl.1` backup that then sat beside it — that backup was destroyed on
|
|
219
|
+
2026-09-19, so the counts below are a record, not something this repo's ledger
|
|
220
|
+
still reproduces.
|
|
221
|
+
|
|
222
|
+
**Read the ledger through `readEventsWithRotations(repoRoot, opts)`**
|
|
223
|
+
(`scripts/lib/events.mjs`), never by `cat`-ing rotation files together. It
|
|
224
|
+
returns the active file plus every archive in time order, counts unreadable
|
|
225
|
+
lines (`malformed_lines`), and — the reason it exists — reports a rotation
|
|
226
|
+
whose archive is no longer on disk as an entry in `gaps` with `complete:
|
|
227
|
+
false`, instead of silently returning a shorter history. Rotation itself no
|
|
228
|
+
longer writes a `.1`..`.N` ring: since #1401 it renames the active file to
|
|
229
|
+
`_archive/events-<firstTs>_<lastTs>.jsonl` and makes an
|
|
230
|
+
`orchestrator.events.rotated` record naming that archive the first line of the
|
|
231
|
+
new active file (`scripts/lib/events-rotation.mjs:230-258`); the reader still
|
|
232
|
+
reads a legacy ring left on disk by older versions
|
|
233
|
+
(`scripts/lib/events.mjs:512-539`).
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
node --input-type=module -e '
|
|
237
|
+
import { readEventsWithRotations } from "./scripts/lib/events.mjs";
|
|
238
|
+
const r = readEventsWithRotations(process.cwd());
|
|
239
|
+
console.error(JSON.stringify({ gaps: r.gaps, malformed_lines: r.malformed_lines, complete: r.complete }));
|
|
240
|
+
for (const e of r.events) process.stdout.write(JSON.stringify(e) + "\n");
|
|
241
|
+
' | jq -s '{
|
|
242
|
+
total_stopped: [.[]|select(.event=="orchestrator.agent.stopped")]|length,
|
|
243
|
+
with_wave: [.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null))]|length,
|
|
244
|
+
with_wave_no_agent: [.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null) and ((.agent//"")==""))]|length,
|
|
245
|
+
with_wave_and_agent:[.[]|select(.event=="orchestrator.agent.stopped" and ((.wave//.wave_number)!=null) and ((.agent//"")!=""))]|length }'
|
|
246
|
+
# 2026-09-18 (active + `.jsonl.1`):
|
|
247
|
+
# → { total_stopped: 16438, with_wave: 6553, with_wave_no_agent: 5930, with_wave_and_agent: 623 }
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
i.e. 5930 of 6553 wave-scoped stops (90.5%) are phantoms, against **0** records
|
|
251
|
+
of the `agent.dispatched` type the reader also accepts. The git-based
|
|
252
|
+
`VEL_COMMITS` / `VEL_LINES` recipe above is the SKILL-level computation and is
|
|
253
|
+
independent of the monitor.
|
|
254
|
+
|
|
255
|
+
The `console.error` line is the honesty half of the reading and belongs in every
|
|
256
|
+
re-measurement: a count taken from a ledger whose `gaps` is non-empty is a count
|
|
257
|
+
over an unknown fraction of the history. Its blind spot is bounded and named —
|
|
258
|
+
an archive deleted BEFORE #1401 left no tombstone and is undetectable by
|
|
259
|
+
construction (`scripts/lib/events.mjs:556-561`), which is exactly how the
|
|
260
|
+
53,896-line `.jsonl.1` vanished on 2026-09-19 without any analysis noticing.
|
|
217
261
|
|
|
218
262
|
### Thresholds
|
|
219
263
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: convergence-monitoring
|
|
3
|
-
user-invocable:
|
|
3
|
+
user-invocable: false
|
|
4
4
|
tags: [autoresearch, convergence, loop-control, wave-executor]
|
|
5
5
|
model: haiku
|
|
6
6
|
model-preference: sonnet
|
|
@@ -32,9 +32,17 @@ For harness < 2.1.105 (no monitor support), the skill's manual probes documented
|
|
|
32
32
|
|
|
33
33
|
## When to Invoke
|
|
34
34
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
35
|
+
This skill is `user-invocable: false` — there is no `/convergence-monitoring` slash
|
|
36
|
+
command. It is reached two ways:
|
|
37
|
+
|
|
38
|
+
**Platform-native monitor (primary path, CC 2.1.105+):** the background watcher below
|
|
39
|
+
auto-starts on `wave-executor` invocation and streams verdicts as it tails
|
|
40
|
+
`events.jsonl` — no explicit call needed.
|
|
41
|
+
|
|
42
|
+
**Direct/model invocation (fallback, harness < 2.1.105, or a standalone assessment):**
|
|
43
|
+
the model dispatches this skill by name — reads wave history from
|
|
44
|
+
`.orchestrator/metrics/events.jsonl`, computes all three signals, reports verdict +
|
|
45
|
+
recovery options.
|
|
38
46
|
|
|
39
47
|
**Embedded (loop context):** `/autoresearch` and `wave-executor` invoke this skill after
|
|
40
48
|
each Impl-Core or Impl-Polish wave when `convergence-monitoring: true` is set in Session
|
|
@@ -48,9 +56,10 @@ toward done or oscillating/stalling.
|
|
|
48
56
|
|
|
49
57
|
## Phase 0: Activation Gate
|
|
50
58
|
|
|
51
|
-
### 0.1
|
|
59
|
+
### 0.1 Direct Invocation
|
|
52
60
|
|
|
53
|
-
If invoked
|
|
61
|
+
If invoked directly (not embedded from `wave-executor` or `/autoresearch`), skip to
|
|
62
|
+
Phase 1 unconditionally.
|
|
54
63
|
|
|
55
64
|
### 0.2 Embedded Invocation
|
|
56
65
|
|
package/skills/eval/SKILL.md
CHANGED
|
@@ -15,7 +15,7 @@ args-schema:
|
|
|
15
15
|
- flag: --verify
|
|
16
16
|
description: "Re-evaluate a stored run-id and diff per-dimension for scoring drift (exit 1 on drift)"
|
|
17
17
|
description: >
|
|
18
|
-
Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-
|
|
18
|
+
Use this skill to run an honest session-process evaluation (Standard v1, aiat-llm-eval/1.0) — score the last completed orchestrator session against the pre-registered rubric-v2 dimensions, run /eval, evaluate this session, produce an eval report, or re-verify a stored eval run for reproducibility. Deterministic-first with an optional advisory LLM judge; never produces a global score.
|
|
19
19
|
---
|
|
20
20
|
|
|
21
21
|
> **Platform Note:** State files use the platform's native directory: `.claude/` (Claude Code), `.codex/` (Codex CLI), or `.cursor/` (Cursor IDE). Shared metrics + the eval journal live in `.orchestrator/metrics/`. See `skills/_shared/platform-tools.md`.
|
|
@@ -23,14 +23,17 @@ description: >
|
|
|
23
23
|
# Eval Skill — Session-Process Evaluation (aiat-llm-eval/1.0)
|
|
24
24
|
|
|
25
25
|
On-demand, honest measurement of ONE completed orchestrator session against the
|
|
26
|
-
pre-registered **rubric-
|
|
26
|
+
pre-registered **rubric-v2** check set. The deterministic engine
|
|
27
27
|
(`scripts/eval-session.mjs` → `scripts/lib/eval/engine.mjs`) reads only local
|
|
28
|
-
metrics files (`sessions.jsonl` + `events.jsonl`), scores the
|
|
28
|
+
metrics files (`sessions.jsonl` + `events.jsonl`), scores the six deterministic
|
|
29
29
|
dimensions, appends a `session-eval` record to the journal, and optionally
|
|
30
|
-
renders an HTML report. An opt-in LLM judge overlays
|
|
30
|
+
renders an HTML report. An opt-in LLM judge overlays ONE advisory dimension
|
|
31
|
+
(`instruction-adherence`; `report-quality` was retired in rubric-v2, #1381).
|
|
31
32
|
|
|
32
33
|
The standard this skill implements is [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md);
|
|
33
|
-
the frozen, content-hashed check set is [`skills/eval/rubric-
|
|
34
|
+
the frozen, content-hashed check set is [`skills/eval/rubric-v2.md`](./rubric-v2.md)
|
|
35
|
+
(stored records written before 2026-09-19 carry `rubric-v1` and are read against
|
|
36
|
+
[`rubric-v1.md`](./rubric-v1.md), which is never edited again).
|
|
34
37
|
|
|
35
38
|
## Invocation
|
|
36
39
|
|
|
@@ -43,7 +46,7 @@ Invoked as `/eval [--session <id>] [--no-write] [--verify <run-id>]` with argume
|
|
|
43
46
|
|
|
44
47
|
**On-demand `/eval` runs regardless of `eval.enabled`** — that flag gates only the automatic session-end eval phase (see Phase 1.1).
|
|
45
48
|
|
|
46
|
-
**Seams used:** `scripts/eval-session.mjs` (deterministic CLI) · `runEvalJudge` / `mergeJudgeDimensions` (`scripts/lib/eval/judge.mjs`, opt-in) · `writeEvalReport` (`scripts/lib/eval/report.mjs`) · `appendEvalRecord` (`scripts/lib/eval/sink.mjs`) · the `eval` config block · [`skills/eval/rubric-
|
|
49
|
+
**Seams used:** `scripts/eval-session.mjs` (deterministic CLI) · `runEvalJudge` / `mergeJudgeDimensions` (`scripts/lib/eval/judge.mjs`, opt-in) · `writeEvalReport` (`scripts/lib/eval/report.mjs`) · `appendEvalRecord` (`scripts/lib/eval/sink.mjs`) · the `eval` config block · [`skills/eval/rubric-v2.md`](./rubric-v2.md) (frozen check set).
|
|
47
50
|
|
|
48
51
|
## Posture Contract (load-bearing — read before executing)
|
|
49
52
|
|
|
@@ -52,9 +55,9 @@ Invoked as `/eval [--session <id>] [--no-write] [--verify <run-id>]` with argume
|
|
|
52
55
|
- **Never guess.** Missing source data yields `cannot-determine` (a first-class,
|
|
53
56
|
non-error verdict) with an honest reason — never a fabricated `pass`/`fail`.
|
|
54
57
|
Do NOT "fill in" a missing KPI or infer a gate result the events do not show.
|
|
55
|
-
- **Deterministic before judge.** The
|
|
56
|
-
on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated`
|
|
57
|
-
|
|
58
|
+
- **Deterministic before judge.** The six deterministic dimensions are complete
|
|
59
|
+
on their own. The judge (Phase 3) is opt-in, ADVISORY, and `uncalibrated` —
|
|
60
|
+
never blend a judge verdict into the deterministic tally.
|
|
58
61
|
- **Journal is SSOT; the report is a derived view.** The append-only
|
|
59
62
|
`.orchestrator/metrics/eval.jsonl` is authoritative. The HTML report is
|
|
60
63
|
rebuildable from any stored record and is never authoritative over the journal.
|
|
@@ -148,10 +151,13 @@ node scripts/eval-session.mjs [--session <id>] --json \
|
|
|
148
151
|
On exit `1` (e.g. "no completed session found"), surface the message and stop —
|
|
149
152
|
do not retry with fabricated inputs.
|
|
150
153
|
|
|
151
|
-
Parse the emitted JSON record. It carries `dimensions[]` (
|
|
152
|
-
entries
|
|
153
|
-
|
|
154
|
-
|
|
154
|
+
Parse the emitted JSON record. It carries `dimensions[]` (6 deterministic
|
|
155
|
+
entries — the two reported-only ones, `guard-friction` and `efficiency-kpis`,
|
|
156
|
+
are always `not-applicable`), `kpis{}`, `provenance.rubric_sha256` (the sha256
|
|
157
|
+
of `rubric-v2.md`; `null` means the rubric file was not found and the append
|
|
158
|
+
will fail validation), `model`, `harness`, and `run_id`. Unless `--no-write` was
|
|
159
|
+
passed, the record is already appended to `.orchestrator/metrics/eval.jsonl` by
|
|
160
|
+
the CLI.
|
|
155
161
|
|
|
156
162
|
**Contamination check:** if the human-render/summary reports a peer-overlapped
|
|
157
163
|
window, note it — `verification-evidence` and `gate-health` will read
|
|
@@ -187,9 +193,13 @@ const merged = mergeJudgeDimensions(record, dimensions);
|
|
|
187
193
|
appendEvalRecord(merged, { path: '.orchestrator/metrics/eval.jsonl' });
|
|
188
194
|
```
|
|
189
195
|
|
|
190
|
-
-
|
|
191
|
-
"uncalibrated"` (the schema firewall rejects any other shape). Keep
|
|
192
|
-
visibly separated from the deterministic
|
|
196
|
+
- The judge dimension arrives `advisory: true` + `calibration_status:
|
|
197
|
+
"uncalibrated"` (the schema firewall rejects any other shape). Keep it
|
|
198
|
+
visibly separated from the deterministic six in the summary.
|
|
199
|
+
- `runEvalJudge` pre-computes a `facts` block (`computeRecordFacts`) from the
|
|
200
|
+
deterministic evidence strings and hands it to the judge OUTSIDE the
|
|
201
|
+
untrusted-data fence — anything countable is counted in code, never inferred
|
|
202
|
+
by the model. Do not reimplement that here.
|
|
193
203
|
- If `runEvalJudge` returns a non-ok `status` (e.g. dispatch failed), keep the
|
|
194
204
|
deterministic record as-is and note the judge was unavailable — the
|
|
195
205
|
deterministic evaluation is complete without it.
|
|
@@ -226,20 +236,20 @@ const res = writeEvalReport(record, { generatedAt: new Date().toISOString() });
|
|
|
226
236
|
Emit a compact, honest per-dimension summary. Status lines only — no global score.
|
|
227
237
|
|
|
228
238
|
```
|
|
229
|
-
## /eval — <session_id> (self-evaluation, aiat-llm-eval/1.0 · rubric-
|
|
239
|
+
## /eval — <session_id> (self-evaluation, aiat-llm-eval/1.0 · rubric-v2 · n=1, no CI)
|
|
230
240
|
|
|
231
241
|
Deterministic:
|
|
232
242
|
verification-evidence PASS <one-line evidence>
|
|
233
243
|
plan-fidelity PASS completion_rate=1.0 (score)
|
|
234
244
|
gate-health PASS <one-line evidence>
|
|
235
|
-
process-safety PASS <one-line evidence + guard
|
|
245
|
+
process-safety PASS <one-line evidence + both guard disclosures>
|
|
246
|
+
guard-friction N/A (reported: blocked=… warned=… loop.warning=… <attribution>)
|
|
236
247
|
efficiency-kpis N/A (reported: duration=…s waves=… agents=… tok_in=… tok_out=… carryover=…)
|
|
237
248
|
|
|
238
249
|
Judge (advisory, uncalibrated) [only when eval.judge != off]:
|
|
239
|
-
instruction-adherence <verdict> advisory
|
|
240
|
-
report-quality <verdict> advisory
|
|
250
|
+
instruction-adherence <verdict> advisory (rule <n> applied)
|
|
241
251
|
|
|
242
|
-
cannot-determine: <k> of
|
|
252
|
+
cannot-determine: <k> of 6 deterministic dimensions (<reasons>)
|
|
243
253
|
Report: .orchestrator/eval/reports/<run_id>.html
|
|
244
254
|
Journal: .orchestrator/metrics/eval.jsonl (appended: <yes|--no-write>)
|
|
245
255
|
Re-verify: node scripts/eval-session.mjs --verify <run_id>
|
|
@@ -266,6 +276,11 @@ node scripts/eval-session.mjs --verify <run-id> --json
|
|
|
266
276
|
changed since the record was written — investigate, do not overwrite.
|
|
267
277
|
- `--verify` reproduces the stored model + timestamp verbatim (no env override),
|
|
268
278
|
so a MATCH is a real reproducibility proof of the scoring, not of model output.
|
|
279
|
+
- **A cross-version DRIFT is not a defect.** A stored `rubric-v1` record
|
|
280
|
+
re-scored by today's rubric-v2 engine necessarily differs on `process-safety`
|
|
281
|
+
and reports `present-in-fresh-only: guard-friction`. Read the record's
|
|
282
|
+
`rubric_version` before treating a diff as a regression (#1400 replaces that
|
|
283
|
+
report with an explicit version verdict).
|
|
269
284
|
|
|
270
285
|
---
|
|
271
286
|
|
|
@@ -282,9 +297,9 @@ platform. Only the judge phase needs harness-specific tooling.
|
|
|
282
297
|
`skills/_shared/platform-tools.md` § Agent Dispatch Pattern.
|
|
283
298
|
- **`harness.platform`** on the record is resolved from `$SO_PLATFORM`
|
|
284
299
|
(falls back to `claude-code`) inside the engine — no skill action needed.
|
|
285
|
-
- The deterministic
|
|
286
|
-
available on all platforms; the judge overlay is a Claude-Code-only
|
|
287
|
-
|
|
300
|
+
- The deterministic six dimensions + the HTML report + `--verify` are fully
|
|
301
|
+
available on all platforms; the judge overlay is a Claude-Code-only
|
|
302
|
+
enrichment.
|
|
288
303
|
|
|
289
304
|
---
|
|
290
305
|
|
package/skills/eval/rubric-v1.md
CHANGED
|
@@ -5,6 +5,7 @@
|
|
|
5
5
|
- **Conforms to standard:** `aiat-llm-eval/1.0` — see [`docs/eval/aiat-llm-eval-v1.md`](../../docs/eval/aiat-llm-eval-v1.md)
|
|
6
6
|
- **Reference engine:** [`scripts/lib/eval/engine.mjs`](../../scripts/lib/eval/engine.mjs) (the executable scorers this document mirrors verbatim)
|
|
7
7
|
- **Hash binding:** the sha256 of THIS FILE is written to every record's `provenance.rubric_sha256`.
|
|
8
|
+
- **Superseded for NEW records by [`rubric-v2.md`](./rubric-v2.md) (2026-09-19, issue #1037).** This file stays frozen and authoritative for every record already carrying `rubric_version: "rubric-v1"` — its formulas are the ones those verdicts were produced under and must not be edited. v2 changed exactly two things: `process-safety` no longer fails on `destructive_guard.blocked`, and the guard counts moved to a new reported-only `guard-friction` dimension. The reasoning and the measurement live in rubric-v2 § Änderungen gegenüber v1.
|
|
8
9
|
|
|
9
10
|
> **Pre-Registration (leading principle, standard §1.1).** The checks below are
|
|
10
11
|
> **fixed BEFORE the first scored run executes against this rubric**. This document
|